Answers.Body: <p>In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: precision is the ratio of the number of shared words to the total number of words in the prediction, and recall is the ratio of the number of shared words to the total number of words in the ground truth.</p>
  <p><a href="https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html#F1" rel="nofollow noreferrer">Source</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-02T03:14:55.527Z
Answers.LastActivityDate: 2021-12-22T23:11:17.893Z
Answers.LastEditDate: 2021-12-22T23:11:17.893Z
Answers.UserDisplayName: Chin Hui Chew
Answers.Body: <p>It really depends on what you are looking for your model to do. For example, do false negatives or false positives really cost your research (or your business)? Also, it's very important to consider your label (class) distribution.</p>
  <p>If you just want to achieve the highest accuracy, and you don't have any issue with your class distribution (that I believe you probably don't in your case) then accuracy works pretty well.</p>
  <p>F1 score might be a better option to use if you need to seek a balance between precision and recall and there is an uneven class distribution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-02T11:40:26.390Z
Answers.Text: Thanks for your answer. In my  case I need to compute F1 to compare my model to another one. So I need a clear definition of F1 in case of question/answering tasks like SQuAD dataset. In other words, I need to know how is F1 score calculated for a model trained on SQuAD dataset.
Answers.UserDisplayName: HLeb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T02:50:07.093Z
Answers.LastActivityDate: 2021-12-22T14:45:33.263Z
Answers.LastEditDate: 2021-12-22T14:45:33.263Z
Answers.UserDisplayName: pedrum
Body: <p>I have an NLP model for answer-extraction. So, basically, I have a paragraph and a question as input, and my model extracts the span of the paragraph that corresponds to the answer to the question.</p>
  <p>I need to know how to compute the F1 score for such models. It is the standard metric (along with Exact Match) used in the literature to evaluate question-answering systems.</p>
  
Tags: <natural-language-processing><natural-language-understanding><metric><question-answering>
Title: How is the F1 score calculated in a question-answering system?
UserDisplayName: HLeb
ThreadText: # How is the F1 score calculated in a question-answering system?
  
  (Asked by: username_0 on 2020-07-26)
  
  I have an NLP model for answer-extraction. So, basically, I have a paragraph and a question as input, and my model extracts the span of the paragraph that corresponds to the answer to the question.
  
  I need to know how to compute the F1 score for such models. It is the standard metric (along with Exact Match) used in the literature to evaluate question-answering systems.
  
  **Tags:** natural-language-processing, natural-language-understanding, metric, question-answering
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-04-02. Score: 5
  
  In QA, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: precision is the ratio of the number of shared words to the total number of words in the prediction, and recall is the ratio of the number of shared words to the total number of words in the ground truth.
  
  [Source](https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html#F1)
  
  
  ---
  
  ### Answer by username_2 on 2020-07-27. Score: 0
  
  It really depends on what you are looking for your model to do. For example, do false negatives or false positives really cost your research (or your business)? Also, it's very important to consider your label (class) distribution.
  
  If you just want to achieve the highest accuracy, and you don't have any issue with your class distribution (that I believe you probably don't in your case) then accuracy works pretty well.
  
  F1 score might be a better option to use if you need to seek a balance between precision and recall and there is an uneven class distribution.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. In my  case I need to compute F1 to compare my model to another one. So I need a clear definition of F1 in case of question/answering tasks like SQuAD dataset. In other words, I need to know how is F1 score calculated for a model trained on SQuAD dataset.  
  
  ---
  
  