Answers.Body: <p>Any LLM that exists could easily be modified to be deterministic. At the present, they sample from a probability distribution for the next word. It is a trivial change to make, so that instead of sampling from this distribution, they always pick the word deemed most likely.</p>
  <p>This has nothing to do with running on a GPU vs CPU: the non-determinism is not a function of hardware but software.</p>
  <p>In many LLMs, this tradeoff is governed by a temperature parameter (it is called temperature for historical reasons) that parameterizes how much randomness will be in the LLM.</p>
  <p>Reference: <a href="https://www.linkedin.com/pulse/llm-text-generation-why-determinism-so-hard-achieve-max-bohun-b7ozf#:%7E:text=Theoretically%2C%20setting%20the%20temperature%20parameter,text%20for%20a%20given%20input." rel="nofollow noreferrer">LLM Text Generation: Why is Determinism so Hard to Achieve?</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-07T17:16:02.490Z
Answers.Text: I think OP wants deterministic sampling behavior for any temperature setting (including high temperature). Setting a random seed will do the trick.
Answers.UserDisplayName: Que
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-08T14:16:51.867Z
Answers.Text: yes - any idea why (to my knowledge) no LLM offers to define a randomness seed to make it deterministic? IMO this would have plenty of useful applications.
Answers.UserDisplayName: user599464
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-22T19:25:23.367Z
Answers.Text: I don't think this is correct. You can set temperature to 0 and even set a seed (if available), thereby always selecting the most likely next token, but that won't get you the same output from GPT or Claude.
Answers.UserDisplayName: zcahgg1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-22T19:35:18.820Z
Answers.Text: Here's a link to some discussions on HN https://news.ycombinator.com/item?id=37446724
Answers.UserDisplayName: zcahgg1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-06T22:49:59.733Z
Answers.LastActivityDate: 2024-04-25T13:38:42.483Z
Answers.LastEditDate: 2024-04-25T13:38:42.483Z
Answers.UserDisplayName: chessprogrammer
Answers.Body: <p>LLMs are mathematical functions executed on floating-point calculations.  The software and hardware implementations cause non-deterministic side-effects.</p>
  <p>In most cases, these non-deterministic side-effects are desirable to produce outcomes that converge towards better overall performance.</p>
  <p>That being said removing the non-deterministic behavior is some cases can be incredibly difficult since this behavior may be buried in some framework.  For example, weight initialization is performed by selecting random small values for regularization in neural network implementations.  This is a desirable effect to prevent vanishing or exploding gradients, but that is not to say that those random values couldn't be explicitly reused in some use case if it was absolutely necessary.</p>
  <p>When people talk about using CPU, vs GPU, it typically references computational speed increases but GPUs include a myriad of computational optimizations and some GPU optimizations for picking random numbers do exist and may be used.  There are also floating point anomalies in some computation environments but that should not be attributed towards being &quot;non-deterministic&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-27T16:48:44.263Z
Answers.LastActivityDate: 2024-11-27T16:48:44.263Z
Answers.UserDisplayName: Henry Tseng
Answers.Body: <p>To answer your question &quot;No there arent any&quot;.</p>
  <p>Executing an LLM such as llama.cpp on a CPU may indeed lead to more predictable outcomes due to the way it handles floating-point calculations. To see the difference for yourself, simply obtain a copy of llama.cpp and adjust the temperature and seed parameters as needed, or apply both if relevant, and conduct your tests. The same experiment can be replicated with a model running on a GPU to compare results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-06T19:56:05.747Z
Answers.LastActivityDate: 2024-04-06T19:56:05.747Z
Answers.UserDisplayName: Talha Latif
Body: <p>LLMs are understood to generate non-deterministic outputs.</p>
  <p>Are there LLMs out there that are capable to producing deterministic outputs for any given input given fixed parameters (like e.g temperature)?</p>
  <p>I heard that llama.cpp - if run on a CPU instead of a GPU appears to generate deterministic outputs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-06T22:15:48.823Z
Comments.Text: When LLMs "generate output" they are sampling from a probability distribution. In principle, it should always be possible to get the same samples from a probability distribution. Read more here: https://community.openai.com/t/is-there-a-way-to-set-a-a-random-seed-for-responses-with-temperature-0/4164/3
Comments.UserDisplayName: Que
Tags: <natural-language-processing><large-language-models><model-request>
Title: Are there strictly deterministic LLMs?
UserDisplayName: user599464
ThreadText: # Are there strictly deterministic LLMs?
  
  (Asked by: username_0 on 2023-12-06)
  
  LLMs are understood to generate non-deterministic outputs.
  
  Are there LLMs out there that are capable to producing deterministic outputs for any given input given fixed parameters (like e.g temperature)?
  
  I heard that llama.cpp - if run on a CPU instead of a GPU appears to generate deterministic outputs.
  
  **Tags:** natural-language-processing, large-language-models, model-request
  > **Comments:**
  > **username_1**: When LLMs "generate output" they are sampling from a probability distribution. In principle, it should always be possible to get the same samples from a probability distribution. Read more here: https://community.openai.com/t/is-there-a-way-to-set-a-a-random-seed-for-responses-with-temperature-0/4164/3  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-12-06. Score: 6
  
  Any LLM that exists could easily be modified to be deterministic. At the present, they sample from a probability distribution for the next word. It is a trivial change to make, so that instead of sampling from this distribution, they always pick the word deemed most likely.
  
  This has nothing to do with running on a GPU vs CPU: the non-determinism is not a function of hardware but software.
  
  In many LLMs, this tradeoff is governed by a temperature parameter (it is called temperature for historical reasons) that parameterizes how much randomness will be in the LLM.
  
  Reference: [LLM Text Generation: Why is Determinism so Hard to Achieve?](https://www.linkedin.com/pulse/llm-text-generation-why-determinism-so-hard-achieve-max-bohun-b7ozf#:%7E:text=Theoretically%2C%20setting%20the%20temperature%20parameter,text%20for%20a%20given%20input.)
  
  > **Comments:**
  > **username_1**: I think OP wants deterministic sampling behavior for any temperature setting (including high temperature). Setting a random seed will do the trick.  
  > **username_0**: yes - any idea why (to my knowledge) no LLM offers to define a randomness seed to make it deterministic? IMO this would have plenty of useful applications.  
  > **username_3**: I don't think this is correct. You can set temperature to 0 and even set a seed (if available), thereby always selecting the most likely next token, but that won't get you the same output from GPT or Claude.  
  > **username_3**: Here's a link to some discussions on HN https://news.ycombinator.com/item?id=37446724  
  
  ---
  
  ### Answer by username_4 on 2024-11-27. Score: 1
  
  LLMs are mathematical functions executed on floating-point calculations. The software and hardware implementations cause non-deterministic side-effects.
  
  In most cases, these non-deterministic side-effects are desirable to produce outcomes that converge towards better overall performance.
  
  That being said removing the non-deterministic behavior is some cases can be incredibly difficult since this behavior may be buried in some framework. For example, weight initialization is performed by selecting random small values for regularization in neural network implementations. This is a desirable effect to prevent vanishing or exploding gradients, but that is not to say that those random values couldn't be explicitly reused in some use case if it was absolutely necessary.
  
  When people talk about using CPU, vs GPU, it typically references computational speed increases but GPUs include a myriad of computational optimizations and some GPU optimizations for picking random numbers do exist and may be used. There are also floating point anomalies in some computation environments but that should not be attributed towards being "non-deterministic".
  
  
  ---
  
  ### Answer by username_5 on 2024-04-06. Score: 1
  
  To answer your question "No there arent any".
  
  Executing an LLM such as llama.cpp on a CPU may indeed lead to more predictable outcomes due to the way it handles floating-point calculations. To see the difference for yourself, simply obtain a copy of llama.cpp and adjust the temperature and seed parameters as needed, or apply both if relevant, and conduct your tests. The same experiment can be replicated with a model running on a GPU to compare results.
  
  
  ---
  
  