Answers.Body: <p>It looks like the method they use for training takes a set of candidate hypotheses <span class="math-container">$\mathcal{U}(x)$</span>, along with associated probabilities, and then minimizes the expected value of the cost function over that distribution. Section 3 has the loss function being minimized: </p>
  
  <p><span class="math-container">$ \mathcal{L}_{Risk} = \sum\limits_{u \in \mathcal{U}(x)} cost(t, u) \frac{p(u|x)}{\sum_{u' \in \mathcal{U}(x)} p(u'|x)} $</span>. </p>
  
  <p>One of the cost functions used is <span class="math-container">$1 - \texttt{BLEU}(t, h)$</span>, where <span class="math-container">$t$</span> is the target and <span class="math-container">$h$</span> is the generated hypothesis. I'm not sure where <span class="math-container">$p(u|x)$</span> is coming from, but <span class="math-container">$1 - \mathcal{L}_{Risk}$</span> for the BLEU cost function is probably what they're refering to when they mention Expected BLEU. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T19:38:33.037Z
Answers.LastActivityDate: 2020-05-03T19:38:33.037Z
Answers.UserDisplayName: Ray
Body: <p>Recently I was reading <a href="https://arxiv.org/abs/1909.06694" rel="nofollow noreferrer">a paper</a> based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training with BLEU and SIMILE. </p>
  
  <p>What I'm unable to understand is what is meant by the expected BLEU cost when training with BLEU and SIMILE? Are there any separate cost functions defined for these scores?</p>
  
  <p>I'm attaching the image of the graph.</p>
  
  <p><a href="https://i.sstatic.net/A2m0N.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/A2m0N.png" alt="enter image description here"></a></p>
  
Tags: <natural-language-processing><training><metric><expectation>
Title: What is meant by the expected BLEU cost when training with BLEU and SIMILE?
UserDisplayName: develop97
ThreadText: # What is meant by the expected BLEU cost when training with BLEU and SIMILE?
  
  (Asked by: username_0 on 2020-05-03)
  
  Recently I was reading [a paper](https://arxiv.org/abs/1909.06694) based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training with BLEU and SIMILE.
  
  What I'm unable to understand is what is meant by the expected BLEU cost when training with BLEU and SIMILE? Are there any separate cost functions defined for these scores?
  
  I'm attaching the image of the graph.
  
  [![enter image description here](https://i.sstatic.net/A2m0N.png)](https://i.sstatic.net/A2m0N.png)
  
  **Tags:** natural-language-processing, training, metric, expectation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-03. Score: 1
  
  It looks like the method they use for training takes a set of candidate hypotheses $\mathcal{U}(x)$, along with associated probabilities, and then minimizes the expected value of the cost function over that distribution. Section 3 has the loss function being minimized:
  
  $ \mathcal{L}\_{Risk} = \sum\limits\_{u \in \mathcal{U}(x)} cost(t, u) \frac{p(u|x)}{\sum\_{u' \in \mathcal{U}(x)} p(u'|x)} $.
  
  One of the cost functions used is $1 - \texttt{BLEU}(t, h)$, where $t$ is the target and $h$ is the generated hypothesis. I'm not sure where $p(u|x)$ is coming from, but $1 - \mathcal{L}\_{Risk}$ for the BLEU cost function is probably what they're refering to when they mention Expected BLEU.
  
  
  ---
  
  