Answers.Body: <p>With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data.</p>
  
  <p>For instance, assuming you have <span class="math-container">$M$</span> sentences <span class="math-container">$s_1, \dots, s_M$</span>, each with probability <span class="math-container">$P(s_i)$</span>, the perplexity is <span class="math-container">$$2^{-l},$$</span> where <span class="math-container">$l = \frac{1}{M} \sum P(s_i) \log P(s_i)$</span> for <span class="math-container">$i \in [1 \dots M]$</span>.</p>
  
  <p>Note that while perplexity might be useful to capturing certain aspects of the model, it is by no means perfect, and, even if you are able to reach great perplexity scores, it will not necessarily translate to a good or even working chat bot.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-29T13:48:13.627Z
Answers.LastActivityDate: 2019-02-26T14:54:25.323Z
Answers.LastEditDate: 2019-02-26T14:54:25.323Z
Answers.UserDisplayName: ginge
Answers.Body: <p>For the definition and calculation of perplexity, please refer to <a href="https://stats.stackexchange.com/a/280794/103153">this answer</a>.</p>
  <p>Google proposed a human evaluation metric called Sensibleness and Specificity Average (SSA) which combines two fundamental aspects of a humanlike chatbot: making sense and being specific. And they conducted some experiments and found that perplexity aligns very well with the SSA.</p>
  <p>Here is the explanation in the paper:</p>
  <blockquote>
  <p>Perplexity measures how well the model predicts the test set data; in other words, how accurately it anticipates what people will say next.<br />
  Our results indicate most of the variance in the human metrics can be explained by the test perplexity.</p>
  </blockquote>
  <p>Their experiments showed a very strong correlation between SSA and perplexity(the lower the perplexity the higher the SSA).
  <a href="https://i.sstatic.net/VUmLB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VUmLB.png" alt="enter image description here" /></a></p>
  <p>References:</p>
  <ol>
  <li><a href="https://arxiv.org/pdf/2001.09977.pdf" rel="nofollow noreferrer">Towards a Human-like Open-Domain Chatbot</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-16T03:36:35.793Z
Answers.LastActivityDate: 2021-01-16T03:36:35.793Z
Answers.UserDisplayName: Lerner Zhang
Body: <p>A few papers I have come across say that <a href="https://en.wikipedia.org/wiki/BLEU" rel="nofollow noreferrer">BLEU</a> is not an appropriate evaluation metric for chatbots, so they use the <strong>perplexity</strong>.</p>
  <p>First of all, what is <em>perplexity</em>? How to calculate it? And why is perplexity a good evaluation metric for chatbots?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-31T13:55:43.850Z
Comments.Text: Here's a question on CrossValidated asking what perplexity https://stats.stackexchange.com/q/10302/82135 is.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-29T08:59:44.560Z
Comments.Text: Please refer to this answer: https://stats.stackexchange.com/a/450945/103153
Comments.UserDisplayName: Lerner Zhang
Tags: <natural-language-processing><chat-bots><metric><bleu><perplexity>
Title: Why is the perplexity a good evaluation metric for chatbots?
UserDisplayName: RuiZhang1993
ThreadText: # Why is the perplexity a good evaluation metric for chatbots?
  
  (Asked by: username_0 on 2017-06-01)
  
  A few papers I have come across say that [BLEU](https://en.wikipedia.org/wiki/BLEU) is not an appropriate evaluation metric for chatbots, so they use the **perplexity**.
  
  First of all, what is *perplexity*? How to calculate it? And why is perplexity a good evaluation metric for chatbots?
  
  **Tags:** natural-language-processing, chat-bots, metric, bleu, perplexity
  
  > **Comments:**
  > **username_1**: Here's a question on CrossValidated asking what perplexity https://stats.stackexchange.com/q/10302/82135 is.  
  > **username_2**: Please refer to this answer: https://stats.stackexchange.com/a/450945/103153  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-10-29. Score: 1
  
  With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data.
  
  For instance, assuming you have $M$ sentences $s\_1, \dots, s\_M$, each with probability $P(s\_i)$, the perplexity is $$2^{-l},$$ where $l = \frac{1}{M} \sum P(s\_i) \log P(s\_i)$ for $i \in [1 \dots M]$.
  
  Note that while perplexity might be useful to capturing certain aspects of the model, it is by no means perfect, and, even if you are able to reach great perplexity scores, it will not necessarily translate to a good or even working chat bot.
  
  
  ---
  
  ### Answer by username_2 on 2021-01-16. Score: 0
  
  For the definition and calculation of perplexity, please refer to [this answer](https://stats.stackexchange.com/a/280794/103153).
  
  Google proposed a human evaluation metric called Sensibleness and Specificity Average (SSA) which combines two fundamental aspects of a humanlike chatbot: making sense and being specific. And they conducted some experiments and found that perplexity aligns very well with the SSA.
  
  Here is the explanation in the paper:
  
  > Perplexity measures how well the model predicts the test set data; in other words, how accurately it anticipates what people will say next.  
  > Our results indicate most of the variance in the human metrics can be explained by the test perplexity.
  
  Their experiments showed a very strong correlation between SSA and perplexity(the lower the perplexity the higher the SSA).
  [![enter image description here](https://i.sstatic.net/VUmLB.png)](https://i.sstatic.net/VUmLB.png)
  
  References:
  
  1. [Towards a Human-like Open-Domain Chatbot](https://arxiv.org/pdf/2001.09977.pdf)
  
  
  ---
  
  