Answers.Body: <p>There are other possible metrics, e.g. meteor and BLEURT.
  They compensate some of the basic problems most researchers would like to avoid BLEU for. The downside of not using known metrics is, that your model is even harder to evaluate against other candidates. If you compare to human gold standard corpus, you should not count on BLEURT too much since it is actually intended to evaluate two systems against a gold corpus and tell you which is better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T11:46:39.003Z
Answers.LastActivityDate: 2021-09-20T11:46:39.003Z
Answers.UserDisplayName: Langueneers
Body: <p>I have a text generation model and I want to evaluate its output by comparing it to a set of gold human-annotated references.</p>
  <p>I went through machine-translation metrics and I found that BLEU is used as the main metric usually.
  I didn't like using it because it's shallow as it uses ngrams comparison; the semantics of the translation is missed.</p>
  <p>Is there any other metric to do a semantic-based evaluation?</p>
  <p>I've thought of using a text similarity model to evaluate the output or even an NLI (Natural language inference) system. I am not sure how precise the evaluation will be because SOTA systems are not really accurate.</p>
  
Tags: <natural-language-processing><machine-translation><text-generation>
Title: Semantic-based evaluation of translations instead of BLEU
UserDisplayName: Minions
ThreadText: # Semantic-based evaluation of translations instead of BLEU
  
  (Asked by: username_0 on 2021-07-07)
  
  I have a text generation model and I want to evaluate its output by comparing it to a set of gold human-annotated references.
  
  I went through machine-translation metrics and I found that BLEU is used as the main metric usually.
  I didn't like using it because it's shallow as it uses ngrams comparison; the semantics of the translation is missed.
  
  Is there any other metric to do a semantic-based evaluation?
  
  I've thought of using a text similarity model to evaluate the output or even an NLI (Natural language inference) system. I am not sure how precise the evaluation will be because SOTA systems are not really accurate.
  
  **Tags:** natural-language-processing, machine-translation, text-generation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-20. Score: 0
  
  There are other possible metrics, e.g. meteor and BLEURT.
  They compensate some of the basic problems most researchers would like to avoid BLEU for. The downside of not using known metrics is, that your model is even harder to evaluate against other candidates. If you compare to human gold standard corpus, you should not count on BLEURT too much since it is actually intended to evaluate two systems against a gold corpus and tell you which is better.
  
  
  ---
  
  