This is a questionnaire to collect human annotations regarding response generation in task-oriented dialogues, whose focus is to achieve a certain goal, such as a hotel reservation or a train booking.
Throughout the questionnaire, you will be asked to rate 6 possible agent answers given a context, in multiple domains of evaluation: "Grammaticality", "Informativeness", "Relevance", "Consistency" and "Overall Quality". There is a total of 4 groups of answers to evaluate. Before starting the evaluation, you must carefully read the provided Guidelines, as they are crucial to understand how the evaluation should be performed: https://docs.google.com/document/d/1-CGsRHd9xsZtNTlTuVY1OiJBtKlwlNUeKfHcaS3T3nI/edit?usp=sharing
Feel free to take a look at them while you annotate the dialogues!
Thank you in advance for the support on this master thesis project. Your help is very much appreciated! I would also like to ask you to keep track of the time you take to fill this form.
Without further due, let's start the evaluation :)