Answers.Body: <p>Language models produce a probability distribution over a set of words. You determine the next word by <em>sampling</em> from this distribution. So, determining the next word is stochastic even though the distribution is the same given the initial prompt.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-13T12:36:05.033Z
Answers.Text: Worth mentioning the role of T in adjusting the distribution for sampling
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-21T16:33:52.210Z
Answers.Text: Maybe it's a stupid question but, shoudn't there be a seed to set to make the sampling reproducible?
Answers.UserDisplayName: ggagliano
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-12T23:15:10.727Z
Answers.LastActivityDate: 2021-11-12T23:15:10.727Z
Answers.UserDisplayName: SpiderRico
Body: <p>For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic weighting component in the process.</p>
  <p>However, in Transformers (self-attention based encoder-decoder type architecture to be specific) we get different outputs with the same prompts (assuming <span class="math-container">$T &gt; 0$</span>). This doesn't make sense to me because the set of weights are always static, so the <em>probability distribution</em> produced should be the same; this simple decoding should yield the same output.</p>
  <p>However, in practice, we observe that it is not actually the case.
  Any reasons why?</p>
  
Tags: <deep-learning><transformer><language-model><natural-language-generation><forward-pass>
Title: Why do language models produce different outputs for same prompt?
UserDisplayName: neel g
ThreadText: # Why do language models produce different outputs for same prompt?
  
  (Asked by: username_0 on 2021-11-12)
  
  For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic weighting component in the process.
  
  However, in Transformers (self-attention based encoder-decoder type architecture to be specific) we get different outputs with the same prompts (assuming $T > 0$). This doesn't make sense to me because the set of weights are always static, so the *probability distribution* produced should be the same; this simple decoding should yield the same output.
  
  However, in practice, we observe that it is not actually the case.
  Any reasons why?
  
  **Tags:** deep-learning, transformer, language-model, natural-language-generation, forward-pass
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-12. Score: 5
  
  Language models produce a probability distribution over a set of words. You determine the next word by *sampling* from this distribution. So, determining the next word is stochastic even though the distribution is the same given the initial prompt.
  
  > **Comments:**
  > **username_2**: Worth mentioning the role of T in adjusting the distribution for sampling  
  > **username_3**: Maybe it's a stupid question but, shoudn't there be a seed to set to make the sampling reproducible?  
  
  ---
  
  