Answers.Body: <p>Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.</p>
  <p>Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T17:15:15.270Z
Answers.Text: In the first assumption, sentences are fed into the classification model 3 times with different annotations. Could that lead to overfitting?
Answers.UserDisplayName: Ahmad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T03:14:40.467Z
Answers.Text: giving same input with different outputs does not cause overfitting, it creates an internal weighting of how much to trust each value. Depending on data this can also confuse the model.
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T15:42:12.450Z
Answers.LastActivityDate: 2021-03-21T15:42:12.450Z
Answers.UserDisplayName: mshlis
Body: <p>I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.</p>
  <pre><code>sentence = ['I', 'like', 'action', 'movies','!']
  annotator_1 = ['O','O', 'B_A', 'I_A', 'O'] 
  annotator_2 = ['O','O', 'B_A', 'I_A', 'O'] 
  annotator_3 = ['O','O', 'B_A', 'O', 'O']
  </code></pre>
  <p>The labels follow the BIO format. That is, <code>B_A</code> means the beginning of aspect-term (action) and <code>I_A</code> indicates inside of aspect-term (movie).
  Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).</p>
  <p>I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.</p>
  <p><em>Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?</em></p>
  <p>Another question.</p>
  <p>I merged the annotations in one final list of labels as follows:</p>
  <pre><code>final_annotation = ['O','O', 'B_A', 'I_A', 'O']
  </code></pre>
  <p>In this case, the final label is chosen based on the majority of labels among three annotators.</p>
  <p><em>Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?</em></p>
  
Tags: <neural-networks><natural-language-processing><python><sentiment-analysis><named-entity-recognition>
Title: How to train a sequence labeling model with annotations from three annotators?
UserDisplayName: Ahmad
ThreadText: # How to train a sequence labeling model with annotations from three annotators?
  
  (Asked by: username_0 on 2021-03-21)
  
  I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.
  
  ```
  sentence = ['I', 'like', 'action', 'movies','!']
  annotator_1 = ['O','O', 'B_A', 'I_A', 'O'] 
  annotator_2 = ['O','O', 'B_A', 'I_A', 'O'] 
  annotator_3 = ['O','O', 'B_A', 'O', 'O']
  
  ```
  
  The labels follow the BIO format. That is, `B_A` means the beginning of aspect-term (action) and `I_A` indicates inside of aspect-term (movie).
  Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).
  
  I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.
  
  *Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?*
  
  Another question.
  
  I merged the annotations in one final list of labels as follows:
  
  ```
  final_annotation = ['O','O', 'B_A', 'I_A', 'O']
  
  ```
  
  In this case, the final label is chosen based on the majority of labels among three annotators.
  
  *Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?*
  
  **Tags:** neural-networks, natural-language-processing, python, sentiment-analysis, named-entity-recognition
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-21. Score: 1 | Accepted answer
  
  Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.
  
  Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))
  
  > **Comments:**
  > **username_0**: In the first assumption, sentences are fed into the classification model 3 times with different annotations. Could that lead to overfitting?  
  > **username_1**: giving same input with different outputs does not cause overfitting, it creates an internal weighting of how much to trust each value. Depending on data this can also confuse the model.  
  
  ---
  
  