<p>RLHF is just one possibility of fine-tuning for generative LLMs, which is used to align an LLM to human tastes.</p>
<p>However, you could just create a bunch of great data, and fine-tune (take a pretrained models and just slightly adjust it) on this high quality data, without any RLHF involved (see for example what phi-1 has done)</p>

CC BY-SA 4.0
2023-07-28T23:39:36.100Z
2023-07-28T23:39:36.100Z
Alberto
<p>So finetuning (on a supervised dataset) is a) extremely expensive (in terms of data collection and annotation, as well as GPU costs). RLHF gets around this by training a <strong>reward model</strong> as well as a <strong>policy model</strong>. The reward model provides a signal to the generated outputs, and the policy model helps guide the LLM towards correct/aligned decoding during generation time.</p>
<p>Sources:</p>
<ol>
<li><a href="https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/" rel="nofollow noreferrer">https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/</a></li>
<li><a href="https://arxiv.org/pdf/2204.05862.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2204.05862.pdf</a></li>
</ol>

CC BY-SA 4.0
2024-01-15T03:56:10.560Z
Gpu cost is probably the least concern for fine tuning.
John Jiang
CC BY-SA 4.0
2023-09-22T18:27:28.950Z
2023-09-22T18:27:28.950Z
information_interchange
<p>I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the  human preferences and afterward using this reward model to fine-tune.</p>
<p>But if thats the case, when fine-tuning is relevant anymore. When to use what?</p>

<large-language-models><fine-tuning><gpt-4><few-shot-learning><rlhf>
What is the difference betwen fine runing and rlhf for llm?
Exploring
# What is the difference betwen fine runing and rlhf for llm?

(Asked by: username_0 on 2023-07-28)

I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the human preferences and afterward using this reward model to fine-tune.

But if thats the case, when fine-tuning is relevant anymore. When to use what?

**Tags:** large-language-models, fine-tuning, gpt-4, few-shot-learning, rlhf


---

## 2 answers

### Answer by username_1 on 2023-07-28. Score: 2

RLHF is just one possibility of fine-tuning for generative LLMs, which is used to align an LLM to human tastes.

However, you could just create a bunch of great data, and fine-tune (take a pretrained models and just slightly adjust it) on this high quality data, without any RLHF involved (see for example what phi-1 has done)


---

### Answer by username_2 on 2023-09-22. Score: 0

So finetuning (on a supervised dataset) is a) extremely expensive (in terms of data collection and annotation, as well as GPU costs). RLHF gets around this by training a **reward model** as well as a **policy model**. The reward model provides a signal to the generated outputs, and the policy model helps guide the LLM towards correct/aligned decoding during generation time.

Sources:

1. <https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/>
2. <https://arxiv.org/pdf/2204.05862.pdf>

> **Comments:**
> **username_3**: Gpu cost is probably the least concern for fine tuning.  

---

