Answers.Body: <p>RLHF is just one possibility of fine-tuning for generative LLMs, which is used to align an LLM to human tastes.</p>
  <p>However, you could just create a bunch of great data, and fine-tune (take a pretrained models and just slightly adjust it) on this high quality data, without any RLHF involved (see for example what phi-1 has done)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T23:39:36.100Z
Answers.LastActivityDate: 2023-07-28T23:39:36.100Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>So finetuning (on a supervised dataset) is a) extremely expensive (in terms of data collection and annotation, as well as GPU costs). RLHF gets around this by training a <strong>reward model</strong> as well as a <strong>policy model</strong>. The reward model provides a signal to the generated outputs, and the policy model helps guide the LLM towards correct/aligned decoding during generation time.</p>
  <p>Sources:</p>
  <ol>
  <li><a href="https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/" rel="nofollow noreferrer">https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/</a></li>
  <li><a href="https://arxiv.org/pdf/2204.05862.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2204.05862.pdf</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-15T03:56:10.560Z
Answers.Text: Gpu cost is probably the least concern for fine tuning.
Answers.UserDisplayName: John Jiang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T18:27:28.950Z
Answers.LastActivityDate: 2023-09-22T18:27:28.950Z
Answers.UserDisplayName: information_interchange
Body: <p>I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the  human preferences and afterward using this reward model to fine-tune.</p>
  <p>But if thats the case, when fine-tuning is relevant anymore. When to use what?</p>
  
Tags: <large-language-models><fine-tuning><gpt-4><few-shot-learning><rlhf>
Title: What is the difference betwen fine runing and rlhf for llm?
UserDisplayName: Exploring
ThreadText: # What is the difference betwen fine runing and rlhf for llm?
  
  (Asked by: username_0 on 2023-07-28)
  
  I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the human preferences and afterward using this reward model to fine-tune.
  
  But if thats the case, when fine-tuning is relevant anymore. When to use what?
  
  **Tags:** large-language-models, fine-tuning, gpt-4, few-shot-learning, rlhf
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-07-28. Score: 2
  
  RLHF is just one possibility of fine-tuning for generative LLMs, which is used to align an LLM to human tastes.
  
  However, you could just create a bunch of great data, and fine-tune (take a pretrained models and just slightly adjust it) on this high quality data, without any RLHF involved (see for example what phi-1 has done)
  
  
  ---
  
  ### Answer by username_2 on 2023-09-22. Score: 0
  
  So finetuning (on a supervised dataset) is a) extremely expensive (in terms of data collection and annotation, as well as GPU costs). RLHF gets around this by training a **reward model** as well as a **policy model**. The reward model provides a signal to the generated outputs, and the policy model helps guide the LLM towards correct/aligned decoding during generation time.
  
  Sources:
  
  1. <https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/>
  2. <https://arxiv.org/pdf/2204.05862.pdf>
  
  > **Comments:**
  > **username_3**: Gpu cost is probably the least concern for fine tuning.  
  
  ---
  
  