Answers.Body: <p><strong>TLDR: if you're using a model without instruction-tuning, you'll likely need to do some additional work during inference to make sure that your outputs are coherent <em>regardless of how old your model is</em>. But even on GPT-2 you can get decent generations with the right technique.</strong></p>
  <p>You're correct in that this has a lot to do with instruction-tuning: GPT-2 wasn't trained to give helpful chatbot-style responses, it was trained to generate random web-text. Yes, there are EOS tokens that indicate the end of a sequence, but during pre-training these separate <em>documents</em>, which tend to be pretty long and likely aren't very coherent as chat-bot responses.</p>
  <p><a href="https://discuss.huggingface.co/t/llama-2-generation-config-top-p-0-6/49916/3" rel="nofollow noreferrer">This also means that newer models that aren't instruction-tuned (like Llama-2) will suffer from similar issues</a>, compared to their instruction-tuned variant (Llama-2 chat).</p>
  <h2>Better responses through prompting</h2>
  <p>However, it's important to note that, at least for larger models, training isn't necessary to get this kind of behavior. <a href="https://ai-plans.com/fs/arxiv/arxiv.org_abs_2312.01552v1.pdf" rel="nofollow noreferrer">Lin et al.</a> explore various methods for eliciting helpful chatbot-like responses through zero-shot prompting (instructions + a markdown-style prompt), ICL (providing examples of queries &amp; responses), retrieval augmented ICL (ICL but retrieving examples that are most similar to the query), and their proposed method which adds a system prompt and uses specially-crafted ICL samples.</p>
  <p><a href="https://i.sstatic.net/9omfNuKN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9omfNuKN.png" alt="From Lin et al. (https://ai-plans.com/fs/arxiv/arxiv.org_abs_2312.01552v1.pdf)" /></a></p>
  <p>My <em>guess</em> is that this wouldn't work as well on less powerful models like GPT-2, as they are much worse at learning from context.</p>
  <h2>Repetition penalty</h2>
  <p>@Eponymous mentioned that there are mechanisms to downweigh repetition during sampling, but it's important to add that people have been working on developing inference-time techniques to improve generation quality since GPT-2 (and before neural models, for that matter). e.g., <a href="https://arxiv.org/pdf/1904.09751" rel="nofollow noreferrer">this famous 2019 paper</a>, proposing nucleus sampling (and targeting GPT-2 Large &amp; XL):
  <a href="https://i.sstatic.net/6HNoti7B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6HNoti7B.png" alt="Holtzman et al. (https://arxiv.org/pdf/1904.09751)" /></a></p>
  <p>With the right techniques, you can definitely get these older models to generate coherent text (see Figure 11 in Appendix B in the above paper for some examples of different sampling techniques + the resulting generations).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-17T19:39:36.290Z
Answers.LastActivityDate: 2024-06-17T19:39:36.290Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <ol>
  <li>LLMs have by themself a token used for end of string, so they themself are the one that decide when a generation should stop</li>
  <li>This kind of questions are usually pretty opinionated, but the main reasons are bigger models, and more training data</li>
  </ol>
  <p>However, tuning the length might also be a bias from the RLHF training some model have received</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-21T17:08:51.917Z
Answers.Text: So models are now able to stop by themselves because they can predict the end of string token. But this was already possible with earlier models like GPT2. But you are saying that that one of the reason how they gained the ability to predict that token effectively bigger models with more training data?
Answers.UserDisplayName: Ricu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-20T12:06:53.457Z
Answers.LastActivityDate: 2023-09-20T12:06:53.457Z
Answers.UserDisplayName: Alberto
Body: <p>One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start talking gibberish or repeating itself.</p>
  <p>This behaviour is different when comparing this to older LLMs. For example: GPT2 would usually only stop generating text when it hit the token limit or a predefined stop sequence. Also, it had a much bigger problem with giving repeating answers. Newer models (especially instruction tuned ones) do not suffer from this problems (e.g. llama 2).</p>
  <p><strong>So I have 2 questions: What mechanisms/techniques are used in current language models such that...</strong></p>
  <ol>
  <li>...<strong>the models know when to stop generating text.</strong></li>
  <li>...<strong>the models do not repeat themselfes and stay on topic.</strong></li>
  </ol>
  <p>I suspect it might have alot to do with instruction tuning but I am happy to hear from you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-07T15:52:34.907Z
Comments.Text: Repeating answers were eliminated by specific mechanisms. For example, down-weighting the probability of recently generated words. Apple researchers published a [paper discussing one effort to deal with this](https://arxiv.org/abs/2206.02369).
Comments.UserDisplayName: Eponymous
Tags: <natural-language-processing><chatgpt><large-language-models><language-model><natural-language-generation>
Title: Why do current language models no longer generate to long or short texts?
UserDisplayName: Ricu
ThreadText: # Why do current language models no longer generate to long or short texts?
  
  (Asked by: username_0 on 2023-09-19)
  
  One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start talking gibberish or repeating itself.
  
  This behaviour is different when comparing this to older LLMs. For example: GPT2 would usually only stop generating text when it hit the token limit or a predefined stop sequence. Also, it had a much bigger problem with giving repeating answers. Newer models (especially instruction tuned ones) do not suffer from this problems (e.g. llama 2).
  
  **So I have 2 questions: What mechanisms/techniques are used in current language models such that...**
  
  1. ...**the models know when to stop generating text.**
  2. ...**the models do not repeat themselfes and stay on topic.**
  
  I suspect it might have alot to do with instruction tuning but I am happy to hear from you.
  
  **Tags:** natural-language-processing, chatgpt, large-language-models, language-model, natural-language-generation
  > **Comments:**
  > **username_1**: Repeating answers were eliminated by specific mechanisms. For example, down-weighting the probability of recently generated words. Apple researchers published a [paper discussing one effort to deal with this](https://arxiv.org/abs/2206.02369).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-06-17. Score: 1
  
  **TLDR: if you're using a model without instruction-tuning, you'll likely need to do some additional work during inference to make sure that your outputs are coherent *regardless of how old your model is*. But even on GPT-2 you can get decent generations with the right technique.**
  
  You're correct in that this has a lot to do with instruction-tuning: GPT-2 wasn't trained to give helpful chatbot-style responses, it was trained to generate random web-text. Yes, there are EOS tokens that indicate the end of a sequence, but during pre-training these separate *documents*, which tend to be pretty long and likely aren't very coherent as chat-bot responses.
  
  [This also means that newer models that aren't instruction-tuned (like Llama-2) will suffer from similar issues](https://discuss.huggingface.co/t/llama-2-generation-config-top-p-0-6/49916/3), compared to their instruction-tuned variant (Llama-2 chat).
  
  ## Better responses through prompting
  
  However, it's important to note that, at least for larger models, training isn't necessary to get this kind of behavior. [Lin et al.](https://ai-plans.com/fs/arxiv/arxiv.org_abs_2312.01552v1.pdf) explore various methods for eliciting helpful chatbot-like responses through zero-shot prompting (instructions + a markdown-style prompt), ICL (providing examples of queries & responses), retrieval augmented ICL (ICL but retrieving examples that are most similar to the query), and their proposed method which adds a system prompt and uses specially-crafted ICL samples.
  
  [![From Lin et al. (https://ai-plans.com/fs/arxiv/arxiv.org_abs_2312.01552v1.pdf)](https://i.sstatic.net/9omfNuKN.png)](https://i.sstatic.net/9omfNuKN.png)
  
  My *guess* is that this wouldn't work as well on less powerful models like GPT-2, as they are much worse at learning from context.
  
  ## Repetition penalty
  
  @Eponymous mentioned that there are mechanisms to downweigh repetition during sampling, but it's important to add that people have been working on developing inference-time techniques to improve generation quality since GPT-2 (and before neural models, for that matter). e.g., [this famous 2019 paper](https://arxiv.org/pdf/1904.09751), proposing nucleus sampling (and targeting GPT-2 Large & XL):
  [![Holtzman et al. (https://arxiv.org/pdf/1904.09751)](https://i.sstatic.net/6HNoti7B.png)](https://i.sstatic.net/6HNoti7B.png)
  
  With the right techniques, you can definitely get these older models to generate coherent text (see Figure 11 in Appendix B in the above paper for some examples of different sampling techniques + the resulting generations).
  
  
  ---
  
  ### Answer by username_3 on 2023-09-20. Score: 0
  
  1. LLMs have by themself a token used for end of string, so they themself are the one that decide when a generation should stop
  2. This kind of questions are usually pretty opinionated, but the main reasons are bigger models, and more training data
  
  However, tuning the length might also be a bias from the RLHF training some model have received
  
  > **Comments:**
  > **username_0**: So models are now able to stop by themselves because they can predict the end of string token. But this was already possible with earlier models like GPT2. But you are saying that that one of the reason how they gained the ability to predict that token effectively bigger models with more training data?  
  
  ---
  
  