Answers.Body: <p>Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references.</p>
  <p><a href="https://en.wikipedia.org/wiki/BLEU#Algorithm" rel="nofollow noreferrer">BLEU</a> and <a href="https://towardsdatascience.com/the-ultimate-performance-metric-in-nlp-111df6c64460" rel="nofollow noreferrer">ROGUE</a> both work by comparing a candidate (ie, model output) to reference text (ie, training data). In a translation task (what these metrics are typically used for) this works quite well, as you can normally assume the translation will use a small set of words common among all references, so naturally the candidate should be using these words in <em>some</em> order. The more references you have the more allowance the model has to interpret the translation it's own way and still achieve a relatively high score.</p>
  <p>However, this does not always work. Both algorithms require word boundaries for them to work properly. You can modify your dataset so that words are properly separated for the algorithms to use, but using your example of source code, this becomes very tricky with indentation and more. Additionally, it's very hard to capture the possible outputs. Not only can a variable be reasonably named hundreds of different ways (making it unlikely an exhaustive reference list exists that would capture your models output), the amount of potentially correct code grows extremely quickly the larger the task becomes, meaning the references become increasingly less useful. If you keep the tasks very short, you could potentially find some use from these algorithms, but better metrics exist.</p>
  <p>It's probably best to use these algorithms for what they're meant for - translation. However, there is absolutely no harm in using them in a task anyway, they're very simple algorithms and could still potentially give you some insight.</p>
  <p>(For source code evaluation, you would probably be better actually just running the code and seeing if it completes the task, maybe making your own scoring system. Or you could draw inspiration from something such as <a href="https://arxiv.org/pdf/2107.03374.pdf" rel="nofollow noreferrer">OpenAI's Codex</a> <a href="https://github.com/openai/human-eval" rel="nofollow noreferrer">HumanEval</a>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T06:48:10.137Z
Answers.LastActivityDate: 2021-08-31T06:48:10.137Z
Answers.UserDisplayName: Recessive
Body: <p>Many machine translation metrics such as BLEU or ROUGE are used to evaluate sequence to sequence models where, usually, the sequences are pieces of natural language.</p>
  <p>Is it possible to use these metrics when the dataset is not constituted of natural language sequences? For instance, if the sequences are source code (in some programming language), does it still make sense to use BLEU or ROUGE? How &quot;good&quot; are these metrics in general?</p>
  
Tags: <machine-translation><metric><bleu>
Title: Does it make sense to use BLEU or ROUGE for any machine translation task?
UserDisplayName: Blencer
ThreadText: # Does it make sense to use BLEU or ROUGE for any machine translation task?
  
  (Asked by: username_0 on 2021-08-24)
  
  Many machine translation metrics such as BLEU or ROUGE are used to evaluate sequence to sequence models where, usually, the sequences are pieces of natural language.
  
  Is it possible to use these metrics when the dataset is not constituted of natural language sequences? For instance, if the sequences are source code (in some programming language), does it still make sense to use BLEU or ROUGE? How "good" are these metrics in general?
  
  **Tags:** machine-translation, metric, bleu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-31. Score: 1 | Accepted answer
  
  Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references.
  
  [BLEU](https://en.wikipedia.org/wiki/BLEU#Algorithm) and [ROGUE](https://towardsdatascience.com/the-ultimate-performance-metric-in-nlp-111df6c64460) both work by comparing a candidate (ie, model output) to reference text (ie, training data). In a translation task (what these metrics are typically used for) this works quite well, as you can normally assume the translation will use a small set of words common among all references, so naturally the candidate should be using these words in *some* order. The more references you have the more allowance the model has to interpret the translation it's own way and still achieve a relatively high score.
  
  However, this does not always work. Both algorithms require word boundaries for them to work properly. You can modify your dataset so that words are properly separated for the algorithms to use, but using your example of source code, this becomes very tricky with indentation and more. Additionally, it's very hard to capture the possible outputs. Not only can a variable be reasonably named hundreds of different ways (making it unlikely an exhaustive reference list exists that would capture your models output), the amount of potentially correct code grows extremely quickly the larger the task becomes, meaning the references become increasingly less useful. If you keep the tasks very short, you could potentially find some use from these algorithms, but better metrics exist.
  
  It's probably best to use these algorithms for what they're meant for - translation. However, there is absolutely no harm in using them in a task anyway, they're very simple algorithms and could still potentially give you some insight.
  
  (For source code evaluation, you would probably be better actually just running the code and seeing if it completes the task, maybe making your own scoring system. Or you could draw inspiration from something such as [OpenAI's Codex](https://arxiv.org/pdf/2107.03374.pdf) [HumanEval](https://github.com/openai/human-eval))
  
  
  ---
  
  