Answers.Body: <p>2AFC does not allow for a &quot;not sure&quot; option. The <em>forced</em> part strictly means that the user <em>must</em> select one of the two options. The data collection form design cannot be considered 2AFC, because you allow the user to effectively skip answering or say &quot;I don't know&quot;.</p>
  <p>This doesn't necessarily mean your form design is wrong, just that it is not 2AFC, and to avoid confusion you should not call it that in any documentation or report.</p>
  <blockquote>
  <p>Is such a HIT considered acceptable to collect data from trainers?</p>
  </blockquote>
  <p>That depends on how it will be used. You will need to consider what the &quot;not sure&quot; ground truth data collection means for your model or analysis. You have a few choices:</p>
  <ul>
  <li>Add &quot;not sure&quot; to model outputs, making this a 3 class problem.</li>
  <li>Model &quot;not sure&quot; as <span class="math-container">$p(A)=0.5$</span> in a two class problem. You will want to check that your model supports this, as some optimisers won't work with probabilities even if the underlying maths of the model would support them.</li>
  <li>Do not add &quot;not sure&quot; records to the dataset.</li>
  </ul>
  <p>Each of these choices will have a different impact on how the model behaves, how you report findings, and how you measure success.</p>
  <blockquote>
  <p>The <code>Not sure</code> choice is provided to avoid polluting the data with some random selections.</p>
  </blockquote>
  <p>This is a good rationale, and does not make the data collection &quot;wrong&quot;. However, it prevents it from being 2AFC. If you have specifically been instructed to create a 2AFC data collection, then you should clarify with your manager/tutor/project lead whether that is a strict requirement.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T13:01:09.333Z
Answers.Text: Thanks, I'm studying you answer. It makes sense to me.
Answers.UserDisplayName: Megidd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T11:28:04.907Z
Answers.LastActivityDate: 2024-01-23T11:34:25.480Z
Answers.LastEditDate: 2024-01-23T11:34:25.480Z
Answers.UserDisplayName: Neil Slater
Body: <p>To collect human intelligence data, we intend to use 2-alternative force choice (2AFC). A sample human intelligence task (HIT) is shown below. Can it be considered a 2AFC? Is such a HIT considered acceptable to collect data from trainers?</p>
  <p>Two choices are provided, i.e. <code>A</code> and <code>B</code>. Trainer can pick <code>A</code>, <code>B</code>, or <code>Not sure</code> for multiple metrics. Optionally, the trainer can explain why they picked.</p>
  <p>The <code>Not sure</code> choice is provided to avoid polluting the data with some random selections.</p>
  <p><a href="https://i.sstatic.net/8BcSO.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8BcSO.jpg" alt="2AFC sample" /></a></p>
  
Tags: <training><training-datasets>
Title: Human trainers: collect data by 2-alternative force choice (2AFC)
UserDisplayName: Megidd
ThreadText: # Human trainers: collect data by 2-alternative force choice (2AFC)
  
  (Asked by: username_0 on 2024-01-23)
  
  To collect human intelligence data, we intend to use 2-alternative force choice (2AFC). A sample human intelligence task (HIT) is shown below. Can it be considered a 2AFC? Is such a HIT considered acceptable to collect data from trainers?
  
  Two choices are provided, i.e. `A` and `B`. Trainer can pick `A`, `B`, or `Not sure` for multiple metrics. Optionally, the trainer can explain why they picked.
  
  The `Not sure` choice is provided to avoid polluting the data with some random selections.
  
  [![2AFC sample](https://i.sstatic.net/8BcSO.jpg)](https://i.sstatic.net/8BcSO.jpg)
  
  **Tags:** training, training-datasets
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-23. Score: 5 | Accepted answer
  
  2AFC does not allow for a "not sure" option. The *forced* part strictly means that the user *must* select one of the two options. The data collection form design cannot be considered 2AFC, because you allow the user to effectively skip answering or say "I don't know".
  
  This doesn't necessarily mean your form design is wrong, just that it is not 2AFC, and to avoid confusion you should not call it that in any documentation or report.
  
  > Is such a HIT considered acceptable to collect data from trainers?
  
  That depends on how it will be used. You will need to consider what the "not sure" ground truth data collection means for your model or analysis. You have a few choices:
  
  * Add "not sure" to model outputs, making this a 3 class problem.
  * Model "not sure" as $p(A)=0.5$ in a two class problem. You will want to check that your model supports this, as some optimisers won't work with probabilities even if the underlying maths of the model would support them.
  * Do not add "not sure" records to the dataset.
  
  Each of these choices will have a different impact on how the model behaves, how you report findings, and how you measure success.
  
  > The `Not sure` choice is provided to avoid polluting the data with some random selections.
  
  This is a good rationale, and does not make the data collection "wrong". However, it prevents it from being 2AFC. If you have specifically been instructed to create a 2AFC data collection, then you should clarify with your manager/tutor/project lead whether that is a strict requirement.
  
  > **Comments:**
  > **username_0**: Thanks, I'm studying you answer. It makes sense to me.  
  
  ---
  
  