Answers.Body: <p>Yes, but I have not seen it done with purely negative samples. Often alignment is done using a positive-negative pair. This is done in RLHF, <a href="https://arxiv.org/abs/2305.18290" rel="nofollow noreferrer">DPO</a>, and <a href="https://arxiv.org/abs/2403.07691" rel="nofollow noreferrer">ORPO</a>.</p>
  <p>There are obviously methods that use solely positive samples so it is mathematically tractable to right this in reverse, using solely negative samples. Often the problem with this is that the space of what you don't want is combinatorically large. Methods that use positive-negative pairs work because the positive and negative answers are syntax and grammar similar. This is better explained by the ORPO paper where they show that fine-tuning on positive-only samples actually increases occurrence of negative samples.</p>
  <p>An example of this:</p>
  <p>(Prompt): Why is the sky green?</p>
  <p>(Positive): The sky is not green.</p>
  <p>(Negative): The sky is green because...</p>
  <p>These answers start off quite similar and choose some of the same tokens. Therefore negative-only could actually cause the model to unlearn positive answers as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-07T19:40:41.757Z
Answers.LastActivityDate: 2024-05-07T19:41:59.120Z
Answers.LastEditDate: 2024-05-07T19:41:59.120Z
Answers.UserDisplayName: naston
Body: <p>If I have a negative example from a large language model like</p>
  <pre><code>{&quot;prompt&quot;: &quot;What is the capital of Italy?&quot;, &quot;completion&quot;: &quot;Milan&quot;}
  </code></pre>
  <p>Is there a way for me to use this negative (incorrect) example to finetune an LLM using SFT so that the model is less likely to produce this answer? The correct answer to this question is <code>&quot;Rome&quot;</code>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-30T22:39:49.240Z
Comments.Text: yes but you have to be very carefull... you can use a negative CCE loss, thus maximizing it using that answer as correct answer (it pretty much downweights that answers)... however, you cannot control which other answer is upweighted, so not really sure that this would help
Comments.UserDisplayName: Alberto
Tags: <fine-tuning>
Title: Can you finetune an LLM using negative examples?
UserDisplayName: tSchema
ThreadText: # Can you finetune an LLM using negative examples?
  
  (Asked by: username_0 on 2024-04-30)
  
  If I have a negative example from a large language model like
  {"prompt": "What is the capital of Italy?", "completion": "Milan"}
  
  Is there a way for me to use this negative (incorrect) example to finetune an LLM using SFT so that the model is less likely to produce this answer? The correct answer to this question is "Rome".
  
  **Tags:** fine-tuning
  > **Comments:**
  > **username_1**: yes but you have to be very carefull... you can use a negative CCE loss, thus maximizing it using that answer as correct answer (it pretty much downweights that answers)... however, you cannot control which other answer is upweighted, so not really sure that this would help  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-05-07. Score: 1
  
  Yes, but I have not seen it done with purely negative samples. Often alignment is done using a positive-negative pair. This is done in RLHF, [DPO](https://arxiv.org/abs/2305.18290), and [ORPO](https://arxiv.org/abs/2403.07691).
  
  There are obviously methods that use solely positive samples so it is mathematically tractable to right this in reverse, using solely negative samples. Often the problem with this is that the space of what you don't want is combinatorically large. Methods that use positive-negative pairs work because the positive and negative answers are syntax and grammar similar. This is better explained by the ORPO paper where they show that fine-tuning on positive-only samples actually increases occurrence of negative samples.
  
  An example of this:
  
  (Prompt): Why is the sky green?
  
  (Positive): The sky is not green.
  
  (Negative): The sky is green because...
  
  These answers start off quite similar and choose some of the same tokens. Therefore negative-only could actually cause the model to unlearn positive answers as well.
  
  
  ---
  
  