Answers.Body: <p><strong>TLDR: It is taught to do that during training</strong></p>
  <p>The secret sauce here is not in the architecture but in the fine-tuning part of the training process. The details of which have not been made public yet.</p>
  <p>Large-language models (LLMs), such as ChatGPT, indeed do nothing else than continually predict the next token for a given sequence of tokens (the prompt).</p>
  <p>LLMs are traditionally simply trained to predict this next token in a self-supervised manner on an immensely large amount of data. However, after having done this, there is a second stage to training, namely fine-tuning. The <a href="https://openai.com/blog/chatgpt" rel="nofollow noreferrer">OpenAI ChatGPT Blog</a> states that it does some human-feedback reinforcement learning.</p>
  <h2>Human feedback reinforcement learning</h2>
  <p>In short, human feedback reinforcement learning is nothing else then prompting the model for some input (question), letting it generate several responses, and ordering these answers based on desired output <em>by humans</em>. There are tons of people ranking these answers by hand for fine-tuning these models.</p>
  <p>ChatGPT and other Chat LLMs are thus trained to respond to the prompt</p>
  <blockquote>
  <p>this is the reason that</p>
  </blockquote>
  <p>with</p>
  <blockquote>
  <ol>
  <li>You prompt seems incomplete</li>
  </ol>
  </blockquote>
  <p>as during fine-tuning this was probably ranked higher by humans than</p>
  <blockquote>
  <ol start="2">
  <li>stackexchange is awesome.</li>
  <li>the internet is cool.</li>
  <li>the sun comes up in the morning.</li>
  </ol>
  </blockquote>
  <p>In essence, it is thus simply trained to respond that way. It has nothing to do with the architecture, wrappers or anything of the sort.</p>
  <p>Now this applies much broader to the general concept of chat. The whole fine-tuning process is basically feeding in questions and receiving answers. So the 'chat functionality' is induced into the model through the training process. However, as you can probably still do, it can finish your stories if you ask it to with prompts such as</p>
  <blockquote>
  <p>You are a writer, finishing stories of others. You start working on &quot;There once was...&quot;.</p>
  </blockquote>
  <p>Which is not necessarily a 'chat', but simply next word prediction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-22T09:03:21.973Z
Answers.LastActivityDate: 2023-04-22T09:03:21.973Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>So I read Wolfram's <a href="https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/" rel="nofollow noreferrer">What Is ChatGPT Doing … and Why Does It Work?</a> but it left one really big question in my mind. His summary [if it could be called that!] really emphasizes that the core model is trained to &quot;continue a piece of text that it’s been given&quot;.</p>
  <p>But if I <em>prompt</em> it with something like:</p>
  <blockquote>
  <p>this is the reason that</p>
  </blockquote>
  <p>it usually gives a <em>response</em> like:</p>
  <blockquote>
  <p>I'm sorry, but your message seems to be incomplete. Could you please provide more context or information so that I can understand what you are trying to convey?</p>
  </blockquote>
  <p>And in general, tends to very much give the impression of a <strong>conversation</strong>, &quot;prompt&quot; vs. &quot;response&quot;.</p>
  <p>Architecturally is there a layer within the core LLM that encodes and gets trained on the &quot;conversation&quot; aspects? Or is that done via a separate standalone system, either an independently trained model or some non-ML wrapper logic?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-20T16:56:56.383Z
Comments.Text: [this is somewhat tangentially related to https://ai.stackexchange.com/questions/39048/how-does-exactly-censorship-work-in-chat-gpt but really a much broader question here I think.]
Comments.UserDisplayName: natevw
Tags: <natural-language-understanding><language-model><natural-language-generation>
Title: What puts the "chat" in a system like ChatGPT?
UserDisplayName: natevw
ThreadText: # What puts the "chat" in a system like ChatGPT?
  
  (Asked by: username_0 on 2023-04-20)
  
  So I read Wolfram's [What Is ChatGPT Doing … and Why Does It Work?](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/) but it left one really big question in my mind. His summary [if it could be called that!] really emphasizes that the core model is trained to "continue a piece of text that it’s been given".
  
  But if I *prompt* it with something like:
  
  > this is the reason that
  
  it usually gives a *response* like:
  
  > I'm sorry, but your message seems to be incomplete. Could you please provide more context or information so that I can understand what you are trying to convey?
  
  And in general, tends to very much give the impression of a **conversation**, "prompt" vs. "response".
  
  Architecturally is there a layer within the core LLM that encodes and gets trained on the "conversation" aspects? Or is that done via a separate standalone system, either an independently trained model or some non-ML wrapper logic?
  
  **Tags:** natural-language-understanding, language-model, natural-language-generation
  > **Comments:**
  > **username_0**: [this is somewhat tangentially related to https://ai.stackexchange.com/questions/39048/how-does-exactly-censorship-work-in-chat-gpt but really a much broader question here I think.]  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-22. Score: 3
  
  **TLDR: It is taught to do that during training**
  
  The secret sauce here is not in the architecture but in the fine-tuning part of the training process. The details of which have not been made public yet.
  
  Large-language models (LLMs), such as ChatGPT, indeed do nothing else than continually predict the next token for a given sequence of tokens (the prompt).
  
  LLMs are traditionally simply trained to predict this next token in a self-supervised manner on an immensely large amount of data. However, after having done this, there is a second stage to training, namely fine-tuning. The [OpenAI ChatGPT Blog](https://openai.com/blog/chatgpt) states that it does some human-feedback reinforcement learning.
  
  ## Human feedback reinforcement learning
  
  In short, human feedback reinforcement learning is nothing else then prompting the model for some input (question), letting it generate several responses, and ordering these answers based on desired output *by humans*. There are tons of people ranking these answers by hand for fine-tuning these models.
  
  ChatGPT and other Chat LLMs are thus trained to respond to the prompt
  
  > this is the reason that
  
  with
  
  > 1. You prompt seems incomplete
  
  as during fine-tuning this was probably ranked higher by humans than
  
  > 2. stackexchange is awesome.
  > 3. the internet is cool.
  > 4. the sun comes up in the morning.
  
  In essence, it is thus simply trained to respond that way. It has nothing to do with the architecture, wrappers or anything of the sort.
  
  Now this applies much broader to the general concept of chat. The whole fine-tuning process is basically feeding in questions and receiving answers. So the 'chat functionality' is induced into the model through the training process. However, as you can probably still do, it can finish your stories if you ask it to with prompts such as
  
  > You are a writer, finishing stories of others. You start working on "There once was...".
  
  Which is not necessarily a 'chat', but simply next word prediction.
  
  
  ---
  
  