Answers.Body: <p>The choice of arithmetic mean for averaging perplexities is mainly for simplicity and consistency. It provides a straightforward measure of overall model performance across multiple sequences.</p>
  <p>Each <span class="math-container">$P_k$</span> is already a summary statistic (perplexity) for each sequence, and averaging them gives a global view of how well the model performs on average across multiple sequences.</p>
  <p>In some cases, alternatives to the arithmetic mean of perplexities might be more appropriate. For example, weighted average that assigns weights to sequences based on their length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-16T04:16:55.373Z
Answers.LastActivityDate: 2024-06-16T04:16:55.373Z
Answers.UserDisplayName: Kulin Patel
Body: <p>The perplexity of the <span class="math-container">$i^{th}$</span> token in the <span class="math-container">$k^{th}$</span> sequence is</p>
  <p><span class="math-container">$$
  P_{ki} = \frac{1}{p(t_{ki})}
  $$</span></p>
  <p>The perplexity aggregated for the <span class="math-container">$k^{th}$</span> sequence is then</p>
  <p><span class="math-container">$$
  P_{k} = \left(\prod_{i=1}^N P_{ki}\right)^{1/N} \\
  = \left(\prod_{i=1}^N \frac{1}{p(t_{ki})} \right)^{1/N}
  $$</span></p>
  <p>which is the geometric mean of the perplexities of the tokens. This makes sense as we are essentially taking the multiplicative inverse of the probability that the model got the whole sequence correct.</p>
  <p>Now my question is how to aggregate the perplexities of several sequences. It seems from various places, including the <a href="https://huggingface.co/spaces/evaluate-metric/perplexity" rel="nofollow noreferrer">Hugging Face Tutorial</a>, I see that the prescription is to take the arithmetic mean of the perplexities of sequences</p>
  <p><span class="math-container">$$
  P = \frac{1}{m} \sum_{k=1}^m P_k
  $$</span></p>
  <p>I am not quite understanding what it means to take the average of 1/probabilities. What is this actually capturing?</p>
  
Tags: <natural-language-processing><large-language-models><perplexity>
Title: How are perplexities over multiple instance aggregated?
UserDisplayName: Borun Chowdhury
ThreadText: # How are perplexities over multiple instance aggregated?
  
  (Asked by: username_0 on 2024-06-15)
  
  The perplexity of the $i^{th}$ token in the $k^{th}$ sequence is
  
  $$
  P\_{ki} = \frac{1}{p(t\_{ki})}
  $$
  
  The perplexity aggregated for the $k^{th}$ sequence is then
  
  $$
  P\_{k} = \left(\prod\_{i=1}^N P\_{ki}\right)^{1/N} \\
  = \left(\prod\_{i=1}^N \frac{1}{p(t\_{ki})} \right)^{1/N}
  $$
  
  which is the geometric mean of the perplexities of the tokens. This makes sense as we are essentially taking the multiplicative inverse of the probability that the model got the whole sequence correct.
  
  Now my question is how to aggregate the perplexities of several sequences. It seems from various places, including the [Hugging Face Tutorial](https://huggingface.co/spaces/evaluate-metric/perplexity), I see that the prescription is to take the arithmetic mean of the perplexities of sequences
  
  $$
  P = \frac{1}{m} \sum\_{k=1}^m P\_k
  $$
  
  I am not quite understanding what it means to take the average of 1/probabilities. What is this actually capturing?
  
  **Tags:** natural-language-processing, large-language-models, perplexity
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-16. Score: 0
  
  The choice of arithmetic mean for averaging perplexities is mainly for simplicity and consistency. It provides a straightforward measure of overall model performance across multiple sequences.
  
  Each $P\_k$ is already a summary statistic (perplexity) for each sequence, and averaging them gives a global view of how well the model performs on average across multiple sequences.
  
  In some cases, alternatives to the arithmetic mean of perplexities might be more appropriate. For example, weighted average that assigns weights to sequences based on their length.
  
  
  ---
  
  