Sequence labeling is a fundamental framework for various natural language processing problems including part-of-speech tagging and named entity recognition. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios. In many cases, ground truth labels are costly and time-consuming to collect or even non-existent, while imperfect ones could be easily accessed or transferred from different domains. A typical example is crowdsourced datasets which have multiple annotations for each sentence which may be noisy or incomplete. Additionally, predictions from multiple source models in transfer learning can be seen as a case of multi-source supervision. In this paper, we propose a novel framework named Consensus Network (CONNET) to conduct training with imperfect annotations from multiple sources. It learns the representation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism. Finally, it leads to a model reflecting the consensus among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings.
## Updated review
I have read the rebuttals. The new version of the paper is clearer and the new baseline experiments are a good addition. 
## Original review
This paper presents an approach to train a neural networks-based model for sequence modelling using labels from different sources. The proposed approach explicitly models the annotator and uses an attention model to select the best aggregation method. The model is used on two scenarios: learning with crowd annotation and cross-domain adaptation. For the first scenario, noisy annotators are simulated with models trained on subsets of the data, the proposed model is compared with related works and is shown to achieve the highest f-1 score. For the second scenario, different domains in three NLP tasks are used and the model is shown to yield the best performance.
I think this paper should be accepted, for the following reasons:
- The approach is novel as far as I can tell, and the approach of learning to aggregate labels is significant, as it could also be applied to tasks where inter-annotators agreement is a problem.
- The experiments are convincing and show the potential of the proposed approach. 
- The comparison with related works is thorough.
Detailed comments
- I don't understand the notion of "normalized expertise" in Section 5.4, can the authors briefly describe it in the paper ?
- The paper is not easy to read, for instance the first paragraph of Section 5 contains critical information to understand the experiments, maybe it should be moved the Section 4 and developed more, typically in two subsections "Application to crowd annotation" and "Application to cross-domain" for example.
- Typos:
    - Section 2, 3rd paragraph "for traget corpora" -> "target"
    - Same paragraph: "Yang & Eisenstien (2018) represented" -> "represents" to be consistent
    - Section 4.1: "BiLSTM-CRF" -> "BLSTM-CRF"