Answers.Body: <p>Like with many things, <em>it depends</em> on what you're trying to accomplish.</p>
  <p>The two major parameters you can tweak are:</p>
  <ul>
  <li><a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-temperature" rel="nofollow noreferrer"><strong><code>temperature</code></strong></a> - Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.</li>
  <li><a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-top_p" rel="nofollow noreferrer"><strong><code>top_p</code></strong></a> - An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered</li>
  </ul>
  <p>In terms of some reference values, the forum post on <a href="https://community.openai.com/t/cheat-sheet-mastering-temperature-and-top-p-in-chatgpt-api-a-few-tips-and-tricks-on-controlling-the-creativity-deterministic-output-of-prompt-responses/172683" rel="nofollow noreferrer">Mastering Temperature and Top_p in ChatGPT API</a> provides the following examples:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Use Case</th>
  <th style="text-align: center;">Temp</th>
  <th style="text-align: center;">Top_p</th>
  <th>Description</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Code Generation</td>
  <td style="text-align: center;"><code>0.2</code></td>
  <td style="text-align: center;"><code>0.1</code></td>
  <td>Generates code that adheres to established patterns and conventions. Output is more deterministic and focused. Useful for generating syntactically correct code.</td>
  </tr>
  <tr>
  <td>Creative Writing</td>
  <td style="text-align: center;"><code>0.7</code></td>
  <td style="text-align: center;"><code>0.8</code></td>
  <td>Generates creative and diverse text for storytelling. Output is more exploratory and less constrained by patterns.</td>
  </tr>
  <tr>
  <td>Chatbot Responses</td>
  <td style="text-align: center;"><code>0.5</code></td>
  <td style="text-align: center;"><code>0.5</code></td>
  <td>Generates conversational responses that balance coherence and diversity. Output is more natural and engaging.</td>
  </tr>
  <tr>
  <td>Code Comment Generation</td>
  <td style="text-align: center;"><code>0.3</code></td>
  <td style="text-align: center;"><code>0.2</code></td>
  <td>Generates code comments that are more likely to be concise and relevant. Output is more deterministic and adheres to conventions.</td>
  </tr>
  <tr>
  <td>Data Analysis Scripting</td>
  <td style="text-align: center;"><code>0.2</code></td>
  <td style="text-align: center;"><code>0.1</code></td>
  <td>Generates data analysis scripts that are more likely to be correct and efficient. Output is more deterministic and focused.</td>
  </tr>
  <tr>
  <td>Exploratory Code Writing</td>
  <td style="text-align: center;"><code>0.6</code></td>
  <td style="text-align: center;"><code>0.7</code></td>
  <td>Generates code that explores alternative solutions and creative approaches. Output is less constrained by established patterns.</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>Note</strong>: It is generally recommended that you set <em>either</em> temp or top_p, but <em><strong>not both</strong></em>.  So for the table above, the values can be seen as roughly equivalent ways to achieve the desired outcome.</p>
  <p>Another reason you might include a higher temperature is if the user indicates the first response is wrong and you want to attempt again, but get a different response.  Although re-prompting with the new information can also work as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-01T18:55:53.473Z
Answers.Text: Thanks. I suppose the temperature=0.2 rather than 0 for coding is for alternative solutions when trying again (like the regenerate answer button in the web interface). I am not sure however if the table means T=0.2 OR top_p=0.1 or it means T=0.2 AND top_p=0.1. In the API documentation for `top_p` they mention `We generally recommend altering this or temperature but not both.`, it's mentioned here : https://platform.openai.com/docs/api-reference/chat/create
Answers.UserDisplayName: userrandrand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-01T19:07:54.540Z
Answers.Text: Yeah, should have clarified - I'll update the answer.  The table means you should set one OR the other, but not both
Answers.UserDisplayName: KyleMit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T21:40:59.770Z
Answers.LastActivityDate: 2023-06-01T19:10:56.807Z
Answers.LastEditDate: 2023-06-01T19:10:56.807Z
Answers.UserDisplayName: KyleMit
Answers.Body: <p>Temperature = 0 means deterministic. The same input gives the same output every time.</p>
  <p>If you are fine with fixed outputs, go for temperature = 0, since it's easier to debug.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-29T08:58:18.300Z
Answers.Text: Hi, thanks. I know that T=0 means deterministic. I am more interested in the advantages of adding temperature with the chatgpt 3.5 model. I believe it is recommended to add a little bit of temperature to the gpt 3 model to avoid it repeating itself in some cases. ChatGPT is a less raw model than GPT 3 and is more knowledgeable of what people like and dislike (such as answers with a lot of repetitions). Hence, I am wondering if there is still a reason to add temperature to the ChatGPT model.
Answers.UserDisplayName: userrandrand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-29T15:24:42.037Z
Answers.Text: The only reason I see is that it's weird to have the same exact response every time, in normal conversations. Imagine replying always "Hello there" to my "Hello". Usually some days you would say "Hi" or "Hello" to reply instead. ChatGPT repeating itself seems more a problem which could be handled with repeat penalty.
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-30T04:44:44.930Z
Answers.Text: That's a good point for a conversation indeed. I suppose if these gpt models are ever trained in chess and are equipped with some tree search algorithms or something it could also be good to not have a predictable game also. Those are some benefits of nondeterministic answers. I am still interested in how the type of answers change when the temperature is increased. I might run some statistics to see how answers change. I am not yet sure what to measure other than correctness. Maybe I could also message word frequency and text length or tendency to digress.
Answers.UserDisplayName: userrandrand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T20:35:44.330Z
Answers.Text: `temperature = 0` does not necessarily imply that the output will be deterministic. See the OpenAI docs.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T21:58:57.927Z
Answers.Text: Can you share a link to the relevant documentation page?
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-01T18:41:24.623Z
Answers.Text: @nbro I think Rexcirus was refering to your comment but did not mention your username
Answers.UserDisplayName: userrandrand
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-21T22:40:26.923Z
Answers.Text: A temperature set to 0 [doesn't guarantee](https://stackoverflow.com/q/75946090/395857) a deterministic result.
Answers.UserDisplayName: Franck Dernoncourt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T09:17:12.087Z
Answers.Text: @FranckDernoncourt Besides the fact that the link is for GPT4, I believe that the spirit of temperature=0 is to ensure deterministic output. The fact that this may not happen in practice (e.g. due to floating point roundings, etc.) should be viewed more as a bug in the implementation.
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T09:18:56.873Z
Answers.Text: But it would be nice to have a link to the docs proving or disproving this.
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T13:53:54.153Z
Answers.Text: @Rexcirus the link also mentions gpt 3.5 https://community.openai.com/t/observing-discrepancy-in-completions-with-temperature-0/73380/7 AFAIK the official docs are not mentioning that, but openai employees confirmed that behavior. If you want to reproduce it yourself: [Prompts that reliably result in unreliable completions](https://genai.stackexchange.com/a/379/109)
Answers.UserDisplayName: Franck Dernoncourt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T19:02:52.860Z
Answers.LastActivityDate: 2023-03-28T19:02:52.860Z
Answers.UserDisplayName: Rexcirus
Body: <p>I believe that it is recommended to have a tiny bit of temperature with GPT 3 even for noncreative tasks like 0.2 or something (I am not entirely sure why).</p>
  <p>Last I checked, and if I remember correctly, the examples from openai on their GitHub page use 0 temperature.</p>
  <p>Is there any benefit in choosing a non-zero temperature for the chatgpt API when the query does not request a creative task? If so, are there some categories or examples?</p>
  <p>[EDIT: to make the answer less subjective, perhaps I could ask what are the benefits of increasing the temperature in the chatGPT API]</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-28T07:15:57.937Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Anshuman Kumar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-29T08:50:16.777Z
Comments.Text: @AnshumanKumar The question is fuzzy I agree, it's asking for a list of benefits of increasing the temperature
Comments.UserDisplayName: userrandrand
Tags: <chatgpt><natural-language-generation><large-language-models>
Title: What temperature would you recommend for the chatgpt api?
UserDisplayName: userrandrand
ThreadText: # What temperature would you recommend for the chatgpt api?
  
  (Asked by: username_0 on 2023-03-27)
  
  I believe that it is recommended to have a tiny bit of temperature with GPT 3 even for noncreative tasks like 0.2 or something (I am not entirely sure why).
  
  Last I checked, and if I remember correctly, the examples from openai on their GitHub page use 0 temperature.
  
  Is there any benefit in choosing a non-zero temperature for the chatgpt API when the query does not request a creative task? If so, are there some categories or examples?
  
  [EDIT: to make the answer less subjective, perhaps I could ask what are the benefits of increasing the temperature in the chatGPT API]
  
  **Tags:** chatgpt, natural-language-generation, large-language-models
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  > **username_0**: @username_1 The question is fuzzy I agree, it's asking for a list of benefits of increasing the temperature  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-05-31. Score: 2
  
  Like with many things, *it depends* on what you're trying to accomplish.
  
  The two major parameters you can tweak are:
  
  * [**`temperature`**](https://platform.openai.com/docs/api-reference/chat/create#chat/create-temperature) - Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.
  * [**`top_p`**](https://platform.openai.com/docs/api-reference/chat/create#chat/create-top_p) - An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top\_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered
  
  In terms of some reference values, the forum post on [Mastering Temperature and Top\_p in ChatGPT API](https://community.openai.com/t/cheat-sheet-mastering-temperature-and-top-p-in-chatgpt-api-a-few-tips-and-tricks-on-controlling-the-creativity-deterministic-output-of-prompt-responses/172683) provides the following examples:
  
  | Use Case | Temp | Top\_p | Description |
  | --- | --- | --- | --- |
  | Code Generation | `0.2` | `0.1` | Generates code that adheres to established patterns and conventions. Output is more deterministic and focused. Useful for generating syntactically correct code. |
  | Creative Writing | `0.7` | `0.8` | Generates creative and diverse text for storytelling. Output is more exploratory and less constrained by patterns. |
  | Chatbot Responses | `0.5` | `0.5` | Generates conversational responses that balance coherence and diversity. Output is more natural and engaging. |
  | Code Comment Generation | `0.3` | `0.2` | Generates code comments that are more likely to be concise and relevant. Output is more deterministic and adheres to conventions. |
  | Data Analysis Scripting | `0.2` | `0.1` | Generates data analysis scripts that are more likely to be correct and efficient. Output is more deterministic and focused. |
  | Exploratory Code Writing | `0.6` | `0.7` | Generates code that explores alternative solutions and creative approaches. Output is less constrained by established patterns. |
  
  **Note**: It is generally recommended that you set *either* temp or top\_p, but ***not both***. So for the table above, the values can be seen as roughly equivalent ways to achieve the desired outcome.
  
  Another reason you might include a higher temperature is if the user indicates the first response is wrong and you want to attempt again, but get a different response. Although re-prompting with the new information can also work as well.
  
  > **Comments:**
  > **username_0**: Thanks. I suppose the temperature=0.2 rather than 0 for coding is for alternative solutions when trying again (like the regenerate answer button in the web interface). I am not sure however if the table means T=0.2 OR top_p=0.1 or it means T=0.2 AND top_p=0.1. In the API documentation for `top_p` they mention `We generally recommend altering this or temperature but not both.`, it's mentioned here : https://platform.openai.com/docs/api-reference/chat/create  
  > **username_2**: Yeah, should have clarified - I'll update the answer.  The table means you should set one OR the other, but not both  
  
  ---
  
  ### Answer by username_3 on 2023-03-28. Score: 0
  
  Temperature = 0 means deterministic. The same input gives the same output every time.
  
  If you are fine with fixed outputs, go for temperature = 0, since it's easier to debug.
  
  > **Comments:**
  > **username_0**: Hi, thanks. I know that T=0 means deterministic. I am more interested in the advantages of adding temperature with the chatgpt 3.5 model. I believe it is recommended to add a little bit of temperature to the gpt 3 model to avoid it repeating itself in some cases. ChatGPT is a less raw model than GPT 3 and is more knowledgeable of what people like and dislike (such as answers with a lot of repetitions). Hence, I am wondering if there is still a reason to add temperature to the ChatGPT model.  
  > **username_3**: The only reason I see is that it's weird to have the same exact response every time, in normal conversations. Imagine replying always "Hello there" to my "Hello". Usually some days you would say "Hi" or "Hello" to reply instead. ChatGPT repeating itself seems more a problem which could be handled with repeat penalty.  
  > **username_0**: That's a good point for a conversation indeed. I suppose if these gpt models are ever trained in chess and are equipped with some tree search algorithms or something it could also be good to not have a predictable game also. Those are some benefits of nondeterministic answers. I am still interested in how the type of answers change when the temperature is increased. I might run some statistics to see how answers change. I am not yet sure what to measure other than correctness. Maybe I could also message word frequency and text length or tendency to digress.  
  > **username_4**: `temperature = 0` does not necessarily imply that the output will be deterministic. See the OpenAI docs.  
  > **username_3**: Can you share a link to the relevant documentation page?  
  > **username_0**: @username_4 I think username_3 was refering to your comment but did not mention your username  
  > **username_5**: A temperature set to 0 [doesn't guarantee](https://stackoverflow.com/q/75946090/395857) a deterministic result.  
  > **username_3**: @username_5 Besides the fact that the link is for GPT4, I believe that the spirit of temperature=0 is to ensure deterministic output. The fact that this may not happen in practice (e.g. due to floating point roundings, etc.) should be viewed more as a bug in the implementation.  
  > **username_3**: But it would be nice to have a link to the docs proving or disproving this.  
  > **username_5**: @username_3 the link also mentions gpt 3.5 https://community.openai.com/t/observing-discrepancy-in-completions-with-temperature-0/73380/7 AFAIK the official docs are not mentioning that, but openai employees confirmed that behavior. If you want to reproduce it yourself: [Prompts that reliably result in unreliable completions](https://genai.stackexchange.com/a/379/109)  
  
  ---
  
  