Answers.Body: <p>Both BLEU and METEOR are meant to evaluate the overall translation quality. METEOR shows a slightly better correlation with human judgment than BLEU, however, it relies on n-gram alignment between the translation hypothesis and reference that needs language-specific paraphrase tables. The quality of the table heavily influences the evaluation quality. I think BLEU was preferred because of its simplicity and language independence. Nowadays, there are many evaluation metrics that correlate much better with human judgment (such as <a href="https://github.com/google-research/bleurt" rel="nofollow noreferrer">BLEURT</a>, <a href="https://github.com/Tiiiger/bert_score" rel="nofollow noreferrer">BertScore</a>, or <a href="https://github.com/Unbabel/COMET" rel="nofollow noreferrer">COMET</a>) and they start to be preferred in the MT community (cf. papers: <a href="https://aclanthology.org/2020.acl-main.448/" rel="nofollow noreferrer">Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine ranslation Evaluation Metrics</a>, <a href="https://aclanthology.org/2021.wmt-1.57" rel="nofollow noreferrer">To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation</a>).</p>
  <p>BLEU score is an average of n-gram precisions, weighted by the so-called brevity penalty that penalizes short high-precision, but low-recall hypotheses.</p>
  <p>METEOR computes both precision and recall. Here, the precision corresponds to a proportion of words that are in the hypothesis and are correct. The recall is a ratio of how many words from the translation hypothesis appeared in the hypothesis. To do so, we need to somehow decide which words are correct. This is done by n-gram alignment between the hypothesis and the reference. If there is no exact match between the two, a table of paraphrases is used. To penalize sentences that contain the correct words, but in the wrong order, there is a reordering penalty term that penalizes such sentences.</p>
  <p>Both metrics measure the same aspects of the translation, but slightly differently:</p>
  <ol>
  <li><p>Precision</p>
  <ul>
  <li>BLEU: directly via n-gram precision</li>
  <li>METEOR: directly in the alignment graph</li>
  </ul>
  </li>
  <li><p>Recall</p>
  <ul>
  <li>BLEU: indirectly via including the brevity penalty</li>
  <li>METEOR: directly in the alignment graph</li>
  </ul>
  </li>
  <li><p>Fluency</p>
  <ul>
  <li>BLEU: directly by considering longer n-grams</li>
  <li>METEOR: indirectly from the properties of the alignment graph</li>
  </ul>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-21T08:35:47.880Z
Answers.LastActivityDate: 2022-07-21T08:35:47.880Z
Answers.UserDisplayName: Jindřich
Answers.Body: <p><strong>BLEU</strong> is a widely-used metric for evaluating the quality of a machine translation output by measuring its correlation to reference translations. It is based on n-gram precision.</p>
  <p><a href="https://machinetranslate.org/bleu" rel="nofollow noreferrer">https://machinetranslate.org/bleu</a></p>
  <p><strong>METEOR</strong>, on the other hand, is a more advanced metric that also compares a machine translation output to reference translations, but it also takes into account additional information such as synonyms, word forms, and sentence structure.</p>
  <p><a href="https://machinetranslate.org/meteor" rel="nofollow noreferrer">https://machinetranslate.org/meteor</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-20T12:32:28.810Z
Answers.LastActivityDate: 2023-01-20T12:32:28.810Z
Answers.UserDisplayName: Cecilia Yalangozian
Body: <p>I am trying to understand the concept of evaluating the machine translation evaluation scores.</p>
  <p>I understand how what BLEU score is trying to achieve. It looks into different n-grams like BLEU-1,BLEU-2, BLEU-3, BLEU-4 and try to match with the human written translation.</p>
  <p>However, I can't really understand what METEOR score is for evaluating MT quality. I am trying understand the rationale intuitively. I am already looking into different blog post but can't really figure out.</p>
  <p>How are these two evaluation metrics different and how are they relevant?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-30T04:58:40.333Z
Comments.Text: I heard of BLEU before, but I am not familiar with METEOR. So, by METEOR, you're referring to [https://www.cs.cmu.edu/~alavie/METEOR/pdf/Banerjee-Lavie-2005-METEOR.pdf](https://www.cs.cmu.edu/~alavie/METEOR/pdf/Banerjee-Lavie-2005-METEOR.pdf)?
Comments.UserDisplayName: nbro
Tags: <machine-learning><deep-learning><comparison><machine-translation><bleu>
Title: What are the differences between BLEU and METEOR?
UserDisplayName: Exploring
ThreadText: # What are the differences between BLEU and METEOR?
  
  (Asked by: username_0 on 2022-04-22)
  
  I am trying to understand the concept of evaluating the machine translation evaluation scores.
  
  I understand how what BLEU score is trying to achieve. It looks into different n-grams like BLEU-1,BLEU-2, BLEU-3, BLEU-4 and try to match with the human written translation.
  
  However, I can't really understand what METEOR score is for evaluating MT quality. I am trying understand the rationale intuitively. I am already looking into different blog post but can't really figure out.
  
  How are these two evaluation metrics different and how are they relevant?
  
  **Tags:** machine-learning, deep-learning, comparison, machine-translation, bleu
  
  > **Comments:**
  > **username_1**: I heard of BLEU before, but I am not familiar with METEOR. So, by METEOR, you're referring to [https://www.cs.cmu.edu/~alavie/METEOR/pdf/Banerjee-Lavie-2005-METEOR.pdf](https://www.cs.cmu.edu/~alavie/METEOR/pdf/Banerjee-Lavie-2005-METEOR.pdf)?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-21. Score: 1
  
  Both BLEU and METEOR are meant to evaluate the overall translation quality. METEOR shows a slightly better correlation with human judgment than BLEU, however, it relies on n-gram alignment between the translation hypothesis and reference that needs language-specific paraphrase tables. The quality of the table heavily influences the evaluation quality. I think BLEU was preferred because of its simplicity and language independence. Nowadays, there are many evaluation metrics that correlate much better with human judgment (such as [BLEURT](https://github.com/google-research/bleurt), [BertScore](https://github.com/Tiiiger/bert_score), or [COMET](https://github.com/Unbabel/COMET)) and they start to be preferred in the MT community (cf. papers: [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine ranslation Evaluation Metrics](https://aclanthology.org/2020.acl-main.448/), [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://aclanthology.org/2021.wmt-1.57)).
  
  BLEU score is an average of n-gram precisions, weighted by the so-called brevity penalty that penalizes short high-precision, but low-recall hypotheses.
  
  METEOR computes both precision and recall. Here, the precision corresponds to a proportion of words that are in the hypothesis and are correct. The recall is a ratio of how many words from the translation hypothesis appeared in the hypothesis. To do so, we need to somehow decide which words are correct. This is done by n-gram alignment between the hypothesis and the reference. If there is no exact match between the two, a table of paraphrases is used. To penalize sentences that contain the correct words, but in the wrong order, there is a reordering penalty term that penalizes such sentences.
  
  Both metrics measure the same aspects of the translation, but slightly differently:
  
  1. Precision
  
     * BLEU: directly via n-gram precision
     * METEOR: directly in the alignment graph
  2. Recall
  
     * BLEU: indirectly via including the brevity penalty
     * METEOR: directly in the alignment graph
  3. Fluency
  
     * BLEU: directly by considering longer n-grams
     * METEOR: indirectly from the properties of the alignment graph
  
  
  ---
  
  ### Answer by username_3 on 2023-01-20. Score: 0
  
  **BLEU** is a widely-used metric for evaluating the quality of a machine translation output by measuring its correlation to reference translations. It is based on n-gram precision.
  
  <https://machinetranslate.org/bleu>
  
  **METEOR**, on the other hand, is a more advanced metric that also compares a machine translation output to reference translations, but it also takes into account additional information such as synonyms, word forms, and sentence structure.
  
  <https://machinetranslate.org/meteor>
  
  
  ---
  
  