Answers.Body: <p>That's right, the Attention Layer output is totally deterministic.</p>
  <p>The <strong>temperature</strong> parameter is related to generative tasks (note that this is not the only thing you can do with Attention and the transformers architecture), and it controls how, given the logits (a vector of the same length as the vocabulary <span class="math-container">$V$</span>, the model <strong>samples</strong> one token among all the available tokens.</p>
  <p>Suppose, for simplicity, that we are doing <strong>Causal generation</strong> and we have a vector of logits <span class="math-container">$u$</span> of length <span class="math-container">$V$</span></p>
  <p>If you do not sample and take just the most probable token at each time step (i.e the <span class="math-container">$\arg\max$</span> of the logits), you always end up with the same generation, given a fixed input.</p>
  <p>Otherwise, we can model the distribution of probability over all tokens with a <em>softmax</em> and sample one token from the distribution at each step <span class="math-container">$t$</span>.</p>
  <p><span class="math-container">$$\mathbb{P}(x=v_l | x_{1:t-1}) = \frac{\exp(u_l )}{\sum_{l'}\exp(u'_l)} \qquad \text{where}\; v_l = x_{1:t-1}l \;, \forall l \in V$$</span></p>
  <p>The <em>temperature</em> modifies this distribution by warping the previous distribution, multiplying the terms inside the exponential by a factor <span class="math-container">$t$</span>.</p>
  <p><span class="math-container">$$\mathbb{P}(x=v_l | x_{1:t-1}) = \frac{\exp(u_l/t)}{\sum_{l'}\exp(u'_l/t)} \qquad \text{where}\; v_l = x_{1:t-1}l \;, \forall l \in V$$</span></p>
  <p>For <span class="math-container">$t \rightarrow 1$</span>, the distribution gets <em>sharper</em>, as it tends to a indicator function over the maximum token (the <span class="math-container">$\arg \max$</span>), and for <span class="math-container">$t = 1$</span> it is equivalent to the <em>softmax</em> formula above.</p>
  <p>If instead you choose <span class="math-container">$t &gt; 1$</span>, the probability distribution of predicted tokens will flatten, i.e the distribution will tend to a uniform distribution as <span class="math-container">$t \rightarrow \infty$</span>.</p>
  <p>References: <a href="https://huggingface.co/blog/how-to-generate" rel="nofollow noreferrer">https://huggingface.co/blog/how-to-generate</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-07T16:58:36.460Z
Answers.LastActivityDate: 2024-03-10T09:15:03.617Z
Answers.LastEditDate: 2024-03-10T09:15:03.617Z
Answers.UserDisplayName: Ciodar
Answers.Body: <blockquote>
  <p>Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?</p>
  </blockquote>
  <p>Of course, there is <em>no injected randomness</em> in a regular Transformer model: you have just and encoder-decoder architecture, with positional encoding, multiple multi-head attention blocks and layer normalization.</p>
  <p>The randomness you describe is typical of GPT model family which are generative, being trained to predict the next word token. Basically, you have a distribution over next tokens and you can control the amount of randomness in the sampling by dividing the logits by a temperature <span class="math-container">$\tau$</span>: in the limit if <span class="math-container">$\tau\to\infty$</span> the predicted distribution becames uniform, otherwise if <span class="math-container">$\tau\to 0$</span> it exactly resembles a one-hot encoding (so a <span class="math-container">$1$</span> in a given place, and all zeros elsewhere.)</p>
  <blockquote>
  <p>I'm not sure of is where exactly randomness comes into play.</p>
  </blockquote>
  <p>For example, at the end of the Transformer you have a dense layer which outputs <span class="math-container">$K$</span> logits: unnormalized probabilities over <span class="math-container">$K$</span> tokens. You can define a Categorical distribution from the logits <span class="math-container">$\alpha$</span> (well, you can also define it from normalized probabilities by applying a softmax on top) from which you can sample the &quot;index&quot; of the next token. To control the sampling you introduce the <span class="math-container">$\tau$</span> as follows <span class="math-container">$\text{Categorical}(\alpha / \tau)$</span>. To generate new text you sample from that: say you start from a context <span class="math-container">$C$</span>, you forward the model to get the logits <span class="math-container">$\alpha$</span> and sample a new token <span class="math-container">$t$</span>, then update the context <span class="math-container">$C=C\cup \{t\}$</span> (so that the model also considers the novel predicted word), and repeat the process again.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-07T16:52:05.147Z
Answers.LastActivityDate: 2023-06-07T16:52:05.147Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>The other answers might be correct for the mathematical abstraction of a GPT-like model, but are false for the actual real-world models. On-device forward passes through any big quantized neural network, including state-of-the-art OpenAI models in 2023, <a href="https://web.archive.org/web/20230328011953/https://community.openai.com/t/a-question-on-determinism/8185/2" rel="nofollow noreferrer">are not deterministic</a>. The difference is not large, but in <code>gpt-4-0314</code> with temperature <code>0</code>, you can get a different token sampled every 100 tokens or so.</p>
  <p>Edit (August 2023): What I wrote above is partly correct. The hardware issues explain some of the randomness; however, OpenAI's Chat Completions API models have more variance than the old Completions API models, which is some evidence that randomness in GPT-4 is not explained by hardware issues only.</p>
  <p>The best guess I've seen is that <a href="https://152334h.github.io/blog/non-determinism-in-gpt-4/" rel="nofollow noreferrer">mixture-of-experts routers operate on batches of queries, and are deterministic (up to hardware issues) <em>on the batch level</em>, not on single queries</a>. There is no information leakage between inputs from different users, but the expert choice in each layer depends on what is in the batch, influencing the output.</p>
  <p>Of course, this is speculation, and although the MoE theory seems plausible, it could turn out that GPT-4 inference requires a GPU setup which makes inference less deterministic than on classical dense models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-20T08:14:16.077Z
Answers.LastActivityDate: 2023-08-15T09:11:59.087Z
Answers.LastEditDate: 2023-08-15T09:11:59.087Z
Answers.UserDisplayName: Daniel Paleka
Body: <p>On a high-level <a href="https://www.linkedin.com/pulse/temperature-check-guide-best-chatgpt-feature-youre-using-berkowitz/" rel="nofollow noreferrer">temperature and randomness affect the output of a generative language model</a>:</p>
  <ol>
  <li><p>Lower temperature: Produces more focused, conservative, and consistent responses.</p>
  </li>
  <li><p>Moderate temperature: Strikes a balance between creativity and consistency. This setting can be useful for general content generation, where a blend of accuracy and inventiveness is desired.</p>
  </li>
  <li><p>Higher temperature: Generates more creative, diverse, and unexpected outputs.</p>
  </li>
  </ol>
  <p>What I'm not sure of is <strong>where exactly</strong> randomness (controlled by the temperature) comes into play. I believe to have understood that it's only after a transformer has done its deterministic work, suggesting some probable next words.</p>
  <p>Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T09:07:25.917Z
Comments.Text: Please, next time, put your specific question in the title to avoid ambiguities about what your actual question is. While reading your post, I thought you were asking about the temperature but then, if we continue to read, it turns out that your question is about the attention layers or the layers inside the transformer and in reality has nothing to do with ChatGPT specifically because you're asking about the transformer in general. Again, this is why we need to see your **specific question in the title**. Also create a post with 1 specific question in mind, not multiple or discussion.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T09:09:26.147Z
Comments.Text: Btw, if you want to understand how the temperature works, see [this question](https://ai.stackexchange.com/questions/32477/what-is-the-temperature-in-the-gpt-models). Also, please, next time, before asking a question, check our existing questions. It's often the case that some of our questions can be answered already with a simple search. Thanks for your cooperation.
Comments.UserDisplayName: nbro
Tags: <transformer><attention><randomness><layers>
Title: Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
UserDisplayName: Hans-Peter Stricker
ThreadText: # Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
  
  (Asked by: username_0 on 2023-06-07)
  
  On a high-level [temperature and randomness affect the output of a generative language model](https://www.linkedin.com/pulse/temperature-check-guide-best-chatgpt-feature-youre-using-berkowitz/):
  
  1. Lower temperature: Produces more focused, conservative, and consistent responses.
  2. Moderate temperature: Strikes a balance between creativity and consistency. This setting can be useful for general content generation, where a blend of accuracy and inventiveness is desired.
  3. Higher temperature: Generates more creative, diverse, and unexpected outputs.
  
  What I'm not sure of is **where exactly** randomness (controlled by the temperature) comes into play. I believe to have understood that it's only after a transformer has done its deterministic work, suggesting some probable next words.
  
  Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
  
  **Tags:** transformer, attention, randomness, layers
  
  > **Comments:**
  > **username_1**: Please, next time, put your specific question in the title to avoid ambiguities about what your actual question is. While reading your post, I thought you were asking about the temperature but then, if we continue to read, it turns out that your question is about the attention layers or the layers inside the transformer and in reality has nothing to do with ChatGPT specifically because you're asking about the transformer in general. Again, this is why we need to see your **specific question in the title**. Also create a post with 1 specific question in mind, not multiple or discussion.  
  > **username_1**: Btw, if you want to understand how the temperature works, see [this question](https://ai.stackexchange.com/questions/32477/what-is-the-temperature-in-the-gpt-models). Also, please, next time, before asking a question, check our existing questions. It's often the case that some of our questions can be answered already with a simple search. Thanks for your cooperation.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2023-06-07. Score: 8
  
  > Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
  
  Of course, there is *no injected randomness* in a regular Transformer model: you have just and encoder-decoder architecture, with positional encoding, multiple multi-head attention blocks and layer normalization.
  
  The randomness you describe is typical of GPT model family which are generative, being trained to predict the next word token. Basically, you have a distribution over next tokens and you can control the amount of randomness in the sampling by dividing the logits by a temperature $\tau$: in the limit if $\tau\to\infty$ the predicted distribution becames uniform, otherwise if $\tau\to 0$ it exactly resembles a one-hot encoding (so a $1$ in a given place, and all zeros elsewhere.)
  
  > I'm not sure of is where exactly randomness comes into play.
  
  For example, at the end of the Transformer you have a dense layer which outputs $K$ logits: unnormalized probabilities over $K$ tokens. You can define a Categorical distribution from the logits $\alpha$ (well, you can also define it from normalized probabilities by applying a softmax on top) from which you can sample the "index" of the next token. To control the sampling you introduce the $\tau$ as follows $\text{Categorical}(\alpha / \tau)$. To generate new text you sample from that: say you start from a context $C$, you forward the model to get the logits $\alpha$ and sample a new token $t$, then update the context $C=C\cup \{t\}$ (so that the model also considers the novel predicted word), and repeat the process again.
  
  
  ---
  
  ### Answer by username_2 on 2023-06-07. Score: 3
  
  That's right, the Attention Layer output is totally deterministic.
  
  The **temperature** parameter is related to generative tasks (note that this is not the only thing you can do with Attention and the transformers architecture), and it controls how, given the logits (a vector of the same length as the vocabulary $V$, the model **samples** one token among all the available tokens.
  
  Suppose, for simplicity, that we are doing **Causal generation** and we have a vector of logits $u$ of length $V$
  
  If you do not sample and take just the most probable token at each time step (i.e the $\arg\max$ of the logits), you always end up with the same generation, given a fixed input.
  
  Otherwise, we can model the distribution of probability over all tokens with a *softmax* and sample one token from the distribution at each step $t$.
  
  $$\mathbb{P}(x=v\_l | x\_{1:t-1}) = \frac{\exp(u\_l )}{\sum\_{l'}\exp(u'\_l)} \qquad \text{where}\; v\_l = x\_{1:t-1}l \;, \forall l \in V$$
  
  The *temperature* modifies this distribution by warping the previous distribution, multiplying the terms inside the exponential by a factor $t$.
  
  $$\mathbb{P}(x=v\_l | x\_{1:t-1}) = \frac{\exp(u\_l/t)}{\sum\_{l'}\exp(u'\_l/t)} \qquad \text{where}\; v\_l = x\_{1:t-1}l \;, \forall l \in V$$
  
  For $t \rightarrow 1$, the distribution gets *sharper*, as it tends to a indicator function over the maximum token (the $\arg \max$), and for $t = 1$ it is equivalent to the *softmax* formula above.
  
  If instead you choose $t > 1$, the probability distribution of predicted tokens will flatten, i.e the distribution will tend to a uniform distribution as $t \rightarrow \infty$.
  
  References: <https://huggingface.co/blog/how-to-generate>
  
  
  ---
  
  ### Answer by username_4 on 2023-06-20. Score: 2
  
  The other answers might be correct for the mathematical abstraction of a GPT-like model, but are false for the actual real-world models. On-device forward passes through any big quantized neural network, including state-of-the-art OpenAI models in 2023, [are not deterministic](https://web.archive.org/web/20230328011953/https://community.openai.com/t/a-question-on-determinism/8185/2). The difference is not large, but in `gpt-4-0314` with temperature `0`, you can get a different token sampled every 100 tokens or so.
  
  Edit (August 2023): What I wrote above is partly correct. The hardware issues explain some of the randomness; however, OpenAI's Chat Completions API models have more variance than the old Completions API models, which is some evidence that randomness in GPT-4 is not explained by hardware issues only.
  
  The best guess I've seen is that [mixture-of-experts routers operate on batches of queries, and are deterministic (up to hardware issues) *on the batch level*, not on single queries](https://152334h.github.io/blog/non-determinism-in-gpt-4/). There is no information leakage between inputs from different users, but the expert choice in each layer depends on what is in the batch, influencing the output.
  
  Of course, this is speculation, and although the MoE theory seems plausible, it could turn out that GPT-4 inference requires a GPU setup which makes inference less deterministic than on classical dense models.
  
  
  ---
  
  