Answers.Body: <p>Quoting <a href="https://learn.microsoft.com/en-us/rest/api/cognitiveservices/azureopenaistable/fine-tunes/create?tabs=HTTP" rel="nofollow noreferrer">Microsoft doc</a>:</p>
  <blockquote>
  <p>The weight to use for loss on the prompt tokens. This controls how much the model tries to learn to generate the prompt (as compared to the completion which always has a weight of 1.0), and can add a stabilizing effect to training when completions are short. If prompts are extremely long (relative to completions), it may make sense to reduce this weight so as to avoid over-prioritizing learning the prompt.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T18:46:51.560Z
Answers.Text: Thanks, bu why does the model attempt to learn to generate the prompt in the first place?
Answers.UserDisplayName: milez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T19:35:09.687Z
Answers.Text: @milez think about the original GPT (not the QA version with RLHF)... if you have a text X with 100 initial words and a text Y 100 words long to predict, it might happen that a user puts the first part of X, so your model has to predict the final part of X + Y, so learning also the "input tokens" still have some benefit in that case... if you are in the QA setting, this makes little to no sense, as you probably don't want to learn "half questions"
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T05:22:56.777Z
Answers.Text: I'm trying to get at the architectural feature that makes the model compute the loss over the prompt in the first place. I don't see this feature in other ml systems or neural networks.
Answers.UserDisplayName: milez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T11:52:50.407Z
Answers.Text: @milez because it's intrinsic of the problem, just like you don't have a "intersection over union" loss in NLP but you have it in CV
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T11:54:59.683Z
Answers.Text: @milez take the phrase _"Artificial intelligence is the ability of machines to perform tasks that are typically associated with human intelligence, such as learning and problem-solving"_... now say you give your GPT the first half, and has to predict the second half... you can see that it might happen that a user inserts only the first third of the phrase, so the model has also to learn to generate the piece between the 1/3 and the 1/2 word, and that "prompt token loss" trains the model also to learn that part
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T18:33:13.043Z
Answers.LastActivityDate: 2023-09-10T18:33:13.043Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>This is an old question but I came across it looking for commentary on the same set of thoughts and figured I'd share what I found here as the current responses don't include it. I'm no expert on this topic but I have fine tuned LLMs several times and your intuition that training on the completion and not the prompt would make more sense and lead to a better model when instruction tuning and fine tuning an LLM is correct.</p>
  <p>Here's a blog post that discusses how to implement fine tuning on just the outputs in axolotl (and notes the author's experience that doing so produces a better model/lora): <a href="https://hamel.dev/notes/llm/finetuning/09_template_free.html" rel="nofollow noreferrer">https://hamel.dev/notes/llm/finetuning/09_template_free.html</a></p>
  <p>And here's a screenshot of a twitter post from the co-founder of a startup, OpenPipe, that focuses on fine tuning LLMs, benchmarking your intuition using llm-as-a-judge (which is imperfect but better than just eval loss for example):
  <a href="https://i.sstatic.net/65qePy3B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/65qePy3B.png" alt="Kyle Corbitt tweet benchmarking train_on_outputs: false in axolotl " /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-06T02:10:51.567Z
Answers.LastActivityDate: 2024-07-06T02:10:51.567Z
Answers.UserDisplayName: user1583016
Body: <p>When fine-tuning large language models, which are commonly decoder transformer architectures, sometimes we want to compute the training loss over the entire prompt, sometimes just the completion generated by the model.</p>
  <p>For example, OpenAI fine-tuning API has the <code>prompt_loss_weight</code> parameter <a href="https://platform.openai.com/docs/api-reference/fine-tunes/object#hyperparams-prompt_loss_weight" rel="nofollow noreferrer">1</a></p>
  <p>When training / fine-tuning with Huggingface transformers or Pytorch, it is common practice to set the prompt token labels to the magic number <code>-100</code> if they are ignored by the loss computation.</p>
  <p>My question is: why is this a thing in the first place? What is it about the transformers that makes it compute the loss / predictions for the prompt as well?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-11T04:55:12.930Z
Comments.Text: There is simply no crisp distinction between prompt and response when training these things. Given any chunk of text, you can turn it into many training examples by choosing many different boundaries between "prompt" and "response".
Comments.UserDisplayName: Daniel Wagner
Tags: <training><large-language-models>
Title: Why do transformers compute the loss over the prompt?
UserDisplayName: milez
ThreadText: # Why do transformers compute the loss over the prompt?
  
  (Asked by: username_0 on 2023-09-10)
  
  When fine-tuning large language models, which are commonly decoder transformer architectures, sometimes we want to compute the training loss over the entire prompt, sometimes just the completion generated by the model.
  
  For example, OpenAI fine-tuning API has the `prompt_loss_weight` parameter [1](https://platform.openai.com/docs/api-reference/fine-tunes/object#hyperparams-prompt_loss_weight)
  
  When training / fine-tuning with Huggingface transformers or Pytorch, it is common practice to set the prompt token labels to the magic number `-100` if they are ignored by the loss computation.
  
  My question is: why is this a thing in the first place? What is it about the transformers that makes it compute the loss / predictions for the prompt as well?
  
  **Tags:** training, large-language-models
  
  > **Comments:**
  > **username_1**: There is simply no crisp distinction between prompt and response when training these things. Given any chunk of text, you can turn it into many training examples by choosing many different boundaries between "prompt" and "response".  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-09-10. Score: 4
  
  Quoting [Microsoft doc](https://learn.microsoft.com/en-us/rest/api/cognitiveservices/azureopenaistable/fine-tunes/create?tabs=HTTP):
  
  > The weight to use for loss on the prompt tokens. This controls how much the model tries to learn to generate the prompt (as compared to the completion which always has a weight of 1.0), and can add a stabilizing effect to training when completions are short. If prompts are extremely long (relative to completions), it may make sense to reduce this weight so as to avoid over-prioritizing learning the prompt.
  
  > **Comments:**
  > **username_0**: Thanks, bu why does the model attempt to learn to generate the prompt in the first place?  
  > **username_2**: @username_0 think about the original GPT (not the QA version with RLHF)... if you have a text X with 100 initial words and a text Y 100 words long to predict, it might happen that a user puts the first part of X, so your model has to predict the final part of X + Y, so learning also the "input tokens" still have some benefit in that case... if you are in the QA setting, this makes little to no sense, as you probably don't want to learn "half questions"  
  > **username_0**: I'm trying to get at the architectural feature that makes the model compute the loss over the prompt in the first place. I don't see this feature in other ml systems or neural networks.  
  > **username_2**: @username_0 because it's intrinsic of the problem, just like you don't have a "intersection over union" loss in NLP but you have it in CV  
  > **username_2**: @username_0 take the phrase _"Artificial intelligence is the ability of machines to perform tasks that are typically associated with human intelligence, such as learning and problem-solving"_... now say you give your GPT the first half, and has to predict the second half... you can see that it might happen that a user inserts only the first third of the phrase, so the model has also to learn to generate the piece between the 1/3 and the 1/2 word, and that "prompt token loss" trains the model also to learn that part  
  
  ---
  
  ### Answer by username_3 on 2024-07-06. Score: 0
  
  This is an old question but I came across it looking for commentary on the same set of thoughts and figured I'd share what I found here as the current responses don't include it. I'm no expert on this topic but I have fine tuned LLMs several times and your intuition that training on the completion and not the prompt would make more sense and lead to a better model when instruction tuning and fine tuning an LLM is correct.
  
  Here's a blog post that discusses how to implement fine tuning on just the outputs in axolotl (and notes the author's experience that doing so produces a better model/lora): <https://hamel.dev/notes/llm/finetuning/09_template_free.html>
  
  And here's a screenshot of a twitter post from the co-founder of a startup, OpenPipe, that focuses on fine tuning LLMs, benchmarking your intuition using llm-as-a-judge (which is imperfect but better than just eval loss for example):
  [![Kyle Corbitt tweet benchmarking train_on_outputs: false in axolotl ](https://i.sstatic.net/65qePy3B.png)](https://i.sstatic.net/65qePy3B.png)
  
  
  ---
  
  