Dear Participant,

This survey investigates your comparative judgments on short extractive summaries (around 100 words long), from tweets related to man-made disaster events, generated using four different summarization algorithms. The form is divided into 4 sections, each concerning a different breaking news story. In each section, you will be shown 4 summaries (generated by 4 different systems) that try to capture relevant information about the event as it happened. You will then be asked to mark your choice corresponding to 3 different questions, each of which tries to analyze a specific aspect of the summaries:

1. Occurrences of unfortunate events such as natural or man-made disasters lead to a sudden surge of posts over Twitter. As people, directly or indirectly affected by the incident, try to provide updates and/or opinions about the unfolding situation, a lot of posts (including retweets, or responses to tweets) contain repetitive content. A summarization algorithm should therefore try to reduce such redundant information. The first question asks you to select the summary that contains the least amount of redundant tweets based upon your subjective judgment.


2. As reliable information from authoritative sources become available only with time, several tweets, at the time of posting, contain unverified or rumourous information. The second question asks you to select the summary that contains the maximum no. tweets containing verified information.

NOTE: In order to aid your judgment, we provide a link to a dataset, at the top of each section, containing the Rumour (R)/ Non-rumour (NR) label against each summary tweet. Please note however that these labels were assigned during the unfolding of the event. Hence, the information contained in some tweets with the label 'R' could have actually been verified later. While selecting your choice, we, therefore, request you to verify the facts by reading about the incident (Wikipedia link for each event is provided at the top of the corresponding section for your immediate perusal).


3. Crisis-related tweets can be broadly categorized into (a) Situational: for e.g., providing ground-zero updates regarding casualties, suspects, witnesses, police investigations, affected regions, etc., and (b) Non-situational: for eg., expressing sympathy towards victims, offering help, showing dissent against government measures, etc. While non-situational tweets help in assessing general public sentiments, situational tweets are often more relevant to humanitarian organizations, crisis-responders and family members of those affected. Hence, the third question asks you to select the summary, that, according to you, contains the maximum amount of situational content.

You are requested to meticulously follow the instructions and answer all the questions individually and judiciously.


Clicking on the "Agree" button below indicates the following:
1. You have read the information presented above.
2. You voluntarily agree to participate in this survey.