Answers.Body: <p>In sequence generating models, for vocabulary of size <span class="math-container">$N$</span> (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form:
  <span class="math-container">$$
  \mathrm{softmax} (x_i/T) \quad i = 1, \ldots N,
  $$</span>
  Here <span class="math-container">$T$</span> is the <strong>temperature</strong>. The output of the softmax is the probability that the next token will be the <span class="math-container">$i$</span>-th word in the vocabulary.</p>
  <p>The temperature determines how greedy the generative model is.</p>
  <p>If the temperature is low, the probabilities to sample other but the class with the highest log probability will be small, and the model will probably output the most correct text, but rather boring, with small variation.</p>
  <p>If the temperature is high, the model can output, with rather high probability, other words than those with the highest probability. The generated text will be more diverse, but there is a higher possibility of grammar mistakes and generation of nonsense.</p>
  <p><a href="https://i.sstatic.net/pKVhO.png" rel="noreferrer"><img src="https://i.sstatic.net/pKVhO.png" alt="enter image description here" /></a></p>
  <p>The difference between the low-temperature case (left) and the high-temperature case for the categorical distribution is illustrated in the picture above, where the heights of the bars correspond to probabilities.</p>
  <p><strong>Example</strong></p>
  <p>A good sample is provided in the <a href="https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&amp;a_bid=76564dff" rel="noreferrer">Deep Learning with Python</a> by François Chollet in chapter 12. An extract from the tutorial, refer to this <a href="https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter12_part01_text-generation.ipynb" rel="noreferrer">notebook</a>.</p>
  <pre><code>import numpy as np
  
  tokens_index = dict(enumerate(text_vectorization.get_vocabulary()))
  
  def sample_next(predictions, temperature=1.0):
      predictions = np.asarray(predictions).astype(&quot;float64&quot;)
      predictions = np.log(predictions) / temperature
      exp_preds = np.exp(predictions)
      predictions = exp_preds / np.sum(exp_preds)
      probas = np.random.multinomial(1, predictions, 1)
      return np.argmax(probas)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-21T18:46:22.880Z
Answers.Text: What is the range of the parameter $T$?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-21T19:06:43.480Z
Answers.Text: @nbro $T$ can be any positive number $T \in (0, \infty)$. Reasonable choice dependents on the particular model since different model produce logits with different magnitude. I added the reference to the popular book on DL. Seems like the idea itself is pretty old.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-21T19:16:01.527Z
Answers.Text: Yes, I guess it's an old idea. The concept of a "temperature" also pops up in other contexts, like _simulated annealing_, but I am not sure they have a similar role in both cases. Although you don't mention GPT-3, I suspect that your definition of temperature is what is being used also in that context. It might be worth checking.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-22T13:11:26.447Z
Answers.Text: I guess this is what NovelAI labels as "randomness"
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-10T14:44:13.750Z
Answers.Text: How does T=0 work without causing divide-by-zero errors ? Is there an epsilon in the denominator of the formula to keep this well posed ?
Answers.UserDisplayName: firdaus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-29T03:46:32.713Z
Answers.Text: @firdaus If you read spiridon_the_sun_rotator's comment, Zero is not inclusive.
Answers.UserDisplayName: Saurabh Jain
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T15:40:15.887Z
Answers.Text: @firdaus For the T=0 case, observe that in the limit as T approaches 0, the probability of choosing the token with the highest log-prob tends towards 100%. So to avoid dividing by zero, I imagine inference algorithms would typically treat T=0 as a special case to always choose the most likely token (see e.g. [this line](https://github.com/ggerganov/llama.cpp/blob/4f4bf35f46600441dec2f941e667291eeb9a18d8/common/sampling.cpp#L235) in the llama.cpp implementation).
Answers.UserDisplayName: Jo Liss
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-21T05:32:42.667Z
Answers.LastActivityDate: 2021-11-21T19:03:53.653Z
Answers.LastEditDate: 2021-11-21T19:03:53.653Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>What does the temperature parameter mean when talking about the GPT models?</p>
  <p>I know that a higher temperature value means more randomness, but I want to know how randomness is introduced.</p>
  <p>Does temperature mean we add noise to the weights/activations or do we add randomness when choosing a token in the softmax layer?</p>
  
Tags: <machine-learning><terminology><gpt><language-model><gpt-3>
Title: What is the "temperature" in the GPT models?
UserDisplayName: Tom Dörr
ThreadText: # What is the "temperature" in the GPT models?
  
  (Asked by: username_0 on 2021-11-21)
  
  What does the temperature parameter mean when talking about the GPT models?
  
  I know that a higher temperature value means more randomness, but I want to know how randomness is introduced.
  
  Does temperature mean we add noise to the weights/activations or do we add randomness when choosing a token in the softmax layer?
  
  **Tags:** machine-learning, terminology, gpt, language-model, gpt-3
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-21. Score: 43 | Accepted answer
  
  In sequence generating models, for vocabulary of size $N$ (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form:
  $$
  \mathrm{softmax} (x\_i/T) \quad i = 1, \ldots N,
  $$
  Here $T$ is the **temperature**. The output of the softmax is the probability that the next token will be the $i$-th word in the vocabulary.
  
  The temperature determines how greedy the generative model is.
  
  If the temperature is low, the probabilities to sample other but the class with the highest log probability will be small, and the model will probably output the most correct text, but rather boring, with small variation.
  
  If the temperature is high, the model can output, with rather high probability, other words than those with the highest probability. The generated text will be more diverse, but there is a higher possibility of grammar mistakes and generation of nonsense.
  
  [![enter image description here](https://i.sstatic.net/pKVhO.png)](https://i.sstatic.net/pKVhO.png)
  
  The difference between the low-temperature case (left) and the high-temperature case for the categorical distribution is illustrated in the picture above, where the heights of the bars correspond to probabilities.
  
  **Example**
  
  A good sample is provided in the [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff) by François Chollet in chapter 12. An extract from the tutorial, refer to this [notebook](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter12_part01_text-generation.ipynb).
  
  ```
  import numpy as np
  
  tokens_index = dict(enumerate(text_vectorization.get_vocabulary()))
  
  def sample_next(predictions, temperature=1.0):
      predictions = np.asarray(predictions).astype("float64")
      predictions = np.log(predictions) / temperature
      exp_preds = np.exp(predictions)
      predictions = exp_preds / np.sum(exp_preds)
      probas = np.random.multinomial(1, predictions, 1)
      return np.argmax(probas)
  
  ```
  
  > **Comments:**
  > **username_2**: What is the range of the parameter $T$?  
  > **username_1**: @username_2 $T$ can be any positive number $T \in (0, \infty)$. Reasonable choice dependents on the particular model since different model produce logits with different magnitude. I added the reference to the popular book on DL. Seems like the idea itself is pretty old.  
  > **username_2**: Yes, I guess it's an old idea. The concept of a "temperature" also pops up in other contexts, like _simulated annealing_, but I am not sure they have a similar role in both cases. Although you don't mention GPT-3, I suspect that your definition of temperature is what is being used also in that context. It might be worth checking.  
  > **username_3**: I guess this is what NovelAI labels as "randomness"  
  > **username_4**: How does T=0 work without causing divide-by-zero errors ? Is there an epsilon in the denominator of the formula to keep this well posed ?  
  > **username_5**: @username_4 If you read username_1's comment, Zero is not inclusive.  
  > **username_6**: @username_4 For the T=0 case, observe that in the limit as T approaches 0, the probability of choosing the token with the highest log-prob tends towards 100%. So to avoid dividing by zero, I imagine inference algorithms would typically treat T=0 as a special case to always choose the most likely token (see e.g. [this line](https://github.com/ggerganov/llama.cpp/blob/4f4bf35f46600441dec2f941e667291eeb9a18d8/common/sampling.cpp#L235) in the llama.cpp implementation).  
  
  ---
  
  