The availability of large datasets is essential for progress in coreference and other areas of NLP. Crowdsourcing has proven a viable alternative to expert annotation, offering similar quality for better scalability. However, crowdsourcing require adjudication, and most models of annotation focus on classification tasks where the set of classes is predetermined. This restriction does not apply to anaphoric annotation, where coders relate markables to coreference chains whose number cannot be predefined. This gap was recently covered with the introduction of a mention pair model of anaphoric annotation (MPA). In this work we extend MPA to alleviate the effects of sparsity inherent in some crowdsourcing environments. Specifically, we use a nonparametric partially pooled structure (based on a stick breaking process), fitting jointly with the ability of the annotators hierarchical community profiles. The individual estimates can thus be improved using information about the community when the data is scarce. We show, using a recently published large-scale crowdsourced anaphora dataset, that the proposed model performs better than its unpooled counterpart in conditions of sparsity, and on par when enough observations are available. The model is thus more resilient to different crowdsourcing setups, and, further provides insights into the community of workers. The model is also flexible enough to be used in standard annotation tasks for classification where it registers on par performance with the state of the art.
This paper extends the unpooled mention pair model of annotation (MPA) (Paun et al., 2018b) with the hierarchical priors (e.g., mean and variance) on the ability of the annotators. The proposed method was evaluated on Phrase Detectives 2 corpus, which was annotated by players in a game with a purpose setting for coreference resolution. To control the sparsity of the dataset, the authors split annotations from larger player workloads into smaller batches, and assumed that each batch was produced by a different player. The experimental results show that, when the data is sparse, the proposed method (CommunityMPA) worked better than MPA on Phrase Detectives 2 corpus in terms of mention-pair accuracy, silver-chain quality, and the performance of the state-of-the-art method trained on the aggregated mention pairs. This paper also includes a discussion about the inferred community profiles. The comparison with the traditional approaches that consider communities showed that the proposed method is comparable to the traditional approaches.
I am wondering of the connection between community and sparsity. This study assumes that knowledge of communities (spammers, adversarial, biased, average or high-quality players) would allow to regularize the ability of the annotators towards. In P2, this paper wrote, "This partially pooled structure can prove effective in conditions of sparsity where there are not enough observations to accurately estimate the ability of the annotators in isolation." I have two questions here: why is it effective to consider community for reducing the problem of the sparsity? If the knowledge of communities is useful, why did the advantage of the proposed method disappear in Figures 1, 2, and 3 when we used all data?
In addition, I'm not convinced with the idea of "breaking the larger player workloads into smaller batches" for simulating the sparsity and communities. This treatment introduces quite a few shadow users whose capabilities are exactly the same, and deviates from the reality of the user community. Does this treatment favor the proposed method over MPA more than necessary? I am also wondering why evaluating portions of the dataset where annotations were made by 'sparse' users would not work to highlight the effectiveness of the proposed method for sparse users.
The impact of this paper would be greater if the experimental results could support the importance of modeling user community on the real data. The authors may justify the simulation procedure of the sparsity because 98.67% of Phrase Detectives 2 corpus was annotated by those who produced more than 40 annotations. However, I also think that it is important to show how the proposed method is effective on the real data with sparse annotators. Currently, Table 1 showed no improvement over the conventional methods.
Minor comment
In Section 2.1: It was difficult to separate which part is the base model (MPA) and the novel proposal without reading Paun et al. (2018b).