Answers.Body: <p>TL;DR: Temperature is applied after repetition penalty, so it smoothes out its effect.</p>
  <p>They are basically independent hyper-parameters of the decoding, but applied after each other.</p>
  <p><strong>Higher temperature</strong> makes the output distribution <strong>more uniform</strong>, so you are likely to get more diverse generations, but at the same time, you risk they will not make sense (in an extreme case, you might even get malformed words).</p>
  <p>Language models, especially when undertrained, tend to repeat what was previously generated. To prevent this, (an almost forgotten) <a href="https://arxiv.org/pdf/1909.05858.pdf" rel="nofollow noreferrer">large LM CTRL</a> introduced the repetition penalty that is now implemented in Huggingface Transformers. It is described in an unnumbered equation in Section 4.1 on page 5:</p>
  <p><a href="https://i.sstatic.net/6Km7W.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6Km7W.png" alt="Repetition penalty equation" /></a></p>
  <p>It <strong>discounts</strong> the probability of <strong>tokens that already appeared</strong> in the generated text, making them less likely to appear again.</p>
  <p>Discounting already generated words is implemented as a <a href="https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/generation/logits_process.py#L181" rel="nofollow noreferrer">RepetitionPenaltyLogitsProcessor</a>, and temperature is implemented as <a href="https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/generation/logits_process.py#L161" rel="nofollow noreferrer">TemperatureLogitsWarper</a> in the Transformers library. <strong><a href="https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/generation/utils.py#L2456" rel="nofollow noreferrer">Processors get applied before Warpers</a></strong>. This means that the effect of discounting of already generated tokens gets smoothed by the temperature smoothing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T14:51:52.260Z
Answers.Text: This is not a bad introductory answer, and I'll upvote it, but I was very much hoping for something which notices e.g. that when temperature and repetition penalty are equal in Transformers, then they *cancel each other out!*
Answers.UserDisplayName: Corbin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T17:51:39.850Z
Answers.Text: @Jindfich I'm having trouble seeing how the formula from the paper incorporates previous words.  The formula looks remarkably similar to the formula for when just the temperature is accounted for.  But maybe I'm not interpreting the i and j suffixes correctly?
Answers.UserDisplayName: matsuo_basho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T09:48:11.603Z
Answers.LastActivityDate: 2023-03-13T09:48:11.603Z
Answers.UserDisplayName: Jindřich
Body: <p>I'm trying to demystify my understanding of various decoding parameters. Building on <a href="https://ai.stackexchange.com/a/32478/69194">our understanding of temperature</a>, <strong>how does the repetition penalty interfere with temperature?</strong></p>
  <p>For example, does something special happen when the penalty is approximately equal to temperature, or approximately its reciprocal? Does the penalty need to be offset or biased based on the temperature?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-13T15:55:21.297Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <language-model><encoder-decoder>
Title: How do temperature and repetition penalty interfere?
UserDisplayName: Corbin
ThreadText: # How do temperature and repetition penalty interfere?
  
  (Asked by: username_0 on 2023-03-12)
  
  I'm trying to demystify my understanding of various decoding parameters. Building on [our understanding of temperature](https://ai.stackexchange.com/a/32478/69194), **how does the repetition penalty interfere with temperature?**
  
  For example, does something special happen when the penalty is approximately equal to temperature, or approximately its reciprocal? Does the penalty need to be offset or biased based on the temperature?
  
  **Tags:** language-model, encoder-decoder
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-13. Score: 2 | Accepted answer
  
  TL;DR: Temperature is applied after repetition penalty, so it smoothes out its effect.
  
  They are basically independent hyper-parameters of the decoding, but applied after each other.
  
  **Higher temperature** makes the output distribution **more uniform**, so you are likely to get more diverse generations, but at the same time, you risk they will not make sense (in an extreme case, you might even get malformed words).
  
  Language models, especially when undertrained, tend to repeat what was previously generated. To prevent this, (an almost forgotten) [large LM CTRL](https://arxiv.org/pdf/1909.05858.pdf) introduced the repetition penalty that is now implemented in Huggingface Transformers. It is described in an unnumbered equation in Section 4.1 on page 5:
  
  [![Repetition penalty equation](https://i.sstatic.net/6Km7W.png)](https://i.sstatic.net/6Km7W.png)
  
  It **discounts** the probability of **tokens that already appeared** in the generated text, making them less likely to appear again.
  
  Discounting already generated words is implemented as a [RepetitionPenaltyLogitsProcessor](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/generation/logits_process.py#L181), and temperature is implemented as [TemperatureLogitsWarper](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/generation/logits_process.py#L161) in the Transformers library. **[Processors get applied before Warpers](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/generation/utils.py#L2456)**. This means that the effect of discounting of already generated tokens gets smoothed by the temperature smoothing.
  
  > **Comments:**
  > **username_0**: This is not a bad introductory answer, and I'll upvote it, but I was very much hoping for something which notices e.g. that when temperature and repetition penalty are equal in Transformers, then they *cancel each other out!*  
  > **username_3**: @Jindfich I'm having trouble seeing how the formula from the paper incorporates previous words.  The formula looks remarkably similar to the formula for when just the temperature is accounted for.  But maybe I'm not interpreting the i and j suffixes correctly?  
  
  ---
  
  