what dataset did they use?
Event dataset with news articles
In order to extract event information from text, a machine reading model must learn to accurately read and interpret the ways in which that information is expressed. But it must also, as the human reader must, aggregate numerous individual value hypotheses into a single coherent global analysis, applying global constraints which reflect prior knowledge of the domain. In this work we focus on the task of extracting plane crash event information from clusters of related news articles whose labels are derived via distant supervision. Unlike previous machine reading work, we assume that while most target values will occur frequently in most clusters, they may also be missing or incorrect. We introduce a novel neural architecture to explicitly model the noisy nature of the data and to deal with these aforementioned learning issues. Our models are trained end-to-end and achieve an improvement of more than 12.1 F$_1$ over previous work, despite using far less linguistic annotation. We apply factor graph constraints to promote more coherent event analyses, with belief propagation inference formulated within the transitions of a recurrent neural network. We show this technique additionally improves maximum F$_1$ by up to 2.8 points, resulting in a relative improvement of $50\%$ over the previous state-of-the-art.
Recent work in the area of machine reading has focused on learning in a scenario with perfect information. Whether identifying target entities for simple cloze style queries BIBREF0 , BIBREF1 , or reasoning over short passages of artificially generated text BIBREF2 , short stories BIBREF3 , or children's stories BIBREF4 , these systems all assume that the corresponding text is the unique source of information necessary for answering the query – one that not only contains the answer, but does not contain misleading or otherwise contradictory information. For more practical question answering, where an information retrieval (IR) component must first fetch the set of relevant passages, the text sources will be less reliable and this assumption must be discarded. Text sources may vary in terms of their integrity (whether or not they are intentionally misleading or unreliable), their accuracy (as in the case of news events, where a truthful but outdated article may contain incorrect information), or their relevance to the query. These characteristics necessitate not only the creation of high-precision readers, but also the development of effective strategies for aggregating conflicting stories into a single cohesive account of the event. Additionally, while many question answering systems are designed to extract a single answer to a single query, a user may wish to understand many aspects of a single entity or event. In machine reading, this is akin to pairing each text passage with multiple queries. Modeling each query as an independent prediction can lead to analyses that are incoherent, motivating the need to model the dependencies between queries. We study these problems through the development of a novel machine reading architecture, which we apply to the task of event extraction from news cluster data. We propose a modular architecture which decomposes the task into three fundamental sub-problems: (1) representation INLINEFORM0 scoring, (2) aggregation, and (3) global constraints. Each corresponds to an exchangeable component of our model. We explore a number of choices for these components, with our best configuration improving performance by INLINEFORM1 F INLINEFORM2 , a INLINEFORM3 relative improvement, over the previous state-of-the-art.