<p>The key ingredient is called Reinforcement Learning from Human Feedback (RLHF), that is having humans rate the model answers and use the feedback to guide the model training.</p>
<p><a href="https://i.sstatic.net/LhJur.png" rel="noreferrer"><img src="https://i.sstatic.net/LhJur.png" alt="enter image description here" /></a></p>
<p>The <a href="https://openai.com/blog/chatgpt/" rel="noreferrer">official blog</a> explains this fairly well.</p>

CC BY-SA 4.0
2023-01-02T19:56:45.610Z
2023-01-02T19:56:45.610Z
Rexcirus
<p>I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good understanding of language, is not directly a language model, but a chatbot? Do we know anything about that? I presume that a lot of conversations was needed in order to train it. Did they simply scrape those conversations from the web, and where did they find such conversations in that case?</p>

CC BY-SA 4.0
2022-12-29T10:05:59.517Z
Can you please focus on one model at a time (either LaMDA or ChatGPT), and ask about the other model in a different post? The answers may change from one model to the other, so it's a good idea to focus, otherwise, you may get different answers, for different ways of training the chatbots, which may be what you're looking for, though.
nbro
CC BY-SA 4.0
2023-01-02T01:30:05.193Z
@nbro While I would really be interested in knowing about all promising approaches, I chose to focus on ChatGPT since that's the only chatbot that I've personally tried (in recent times at least).
HelloGoodbye
CC BY-SA 4.0
2023-01-02T03:56:10.093Z
This article describes it well: ["ChatGPT: How Does It Work Internally?"](https://pub.towardsai.net/chatgpt-how-does-it-work-internally-e0b3e23601a1)
Snehal Patel
<natural-language-processing><chat-bots><training-datasets><language-model><chatgpt>
How was ChatGPT trained?
HelloGoodbye
# How was ChatGPT trained?

(Asked by: username_0 on 2022-12-29)

I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good understanding of language, is not directly a language model, but a chatbot? Do we know anything about that? I presume that a lot of conversations was needed in order to train it. Did they simply scrape those conversations from the web, and where did they find such conversations in that case?

**Tags:** natural-language-processing, chat-bots, training-datasets, language-model, chatgpt

> **Comments:**
> **username_1**: Can you please focus on one model at a time (either LaMDA or ChatGPT), and ask about the other model in a different post? The answers may change from one model to the other, so it's a good idea to focus, otherwise, you may get different answers, for different ways of training the chatbots, which may be what you're looking for, though.  
> **username_0**: @username_1 While I would really be interested in knowing about all promising approaches, I chose to focus on ChatGPT since that's the only chatbot that I've personally tried (in recent times at least).  
> **username_2**: This article describes it well: ["ChatGPT: How Does It Work Internally?"](https://pub.towardsai.net/chatgpt-how-does-it-work-internally-e0b3e23601a1)  

---

## 1 answers

### Answer by username_3 on 2023-01-02. Score: 6 | Accepted answer

The key ingredient is called Reinforcement Learning from Human Feedback (RLHF), that is having humans rate the model answers and use the feedback to guide the model training.

[![enter image description here](https://i.sstatic.net/LhJur.png)](https://i.sstatic.net/LhJur.png)

The [official blog](https://openai.com/blog/chatgpt/) explains this fairly well.


---

