Body: <p>I am not technically familiar with AI or neural networks beyond a tech news reading level of knowledge, so I apologise if this is a dumb question.</p>
  <p>I was recently reading <a href="https://arstechnica.com/gadgets/2023/01/the-generative-ai-revolution-has-begun-how-did-we-get-here/3/" rel="nofollow noreferrer">this article</a> on <a href="https://www.arstechnica.com" rel="nofollow noreferrer">Ars Technica</a>.  It is a high level description of the history of generative AI models (it's a very good article, highly recommended).</p>
  <p>When discussing large language models, the following passage appears:</p>
  <blockquote>
  <p>But there was also a surprise. The OpenAI researchers discovered that in making the models bigger, they didn’t just get better at producing text. The models could learn entirely new behaviors simply by being shown new training data. In particular, <strong>the researchers discovered that GPT3 could be trained to follow instructions in plain English without having to explicitly design the model that way.</strong> (emphasis added)</p>
  </blockquote>
  <p>I have three questions:</p>
  <ol>
  <li><p>Was this instruction-following behaviour truly emergent, as in completely unexpected and unplanned for?</p>
  </li>
  <li><p>This seems completely at odds with the usual description of transformer models as transforming text of one kind into another.  Following instructions seems, at a conceptual level, to be something much higher? If this is the case, do we have any idea how this emerged, and what properties of the model it is rooted in?</p>
  </li>
  <li><p>Do we have any idea how wide the scope of this kind of &quot;instruction following&quot; is? I.e. can the models make sense and respond &quot;sensibly&quot; (not correctly - I am not concerned with the correctness of the response, but with its relation to the instruction) to <em>any</em> instruction related to text?  Or are there specific kinds of instructions they are able to comprehend, and others they fail at?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-11T20:11:20.003Z
Comments.Text: I'm not completely au-fait with the topic, so can't really answer it myself, but I think you misunderstand the link.  "it's not truly emergent behaviour, it was specifically trained for" -- what I take away from the link is that it *was* emergent behaviour, but then OpenAI fine tuned their training methods in order to make it better.  Unoptimized GPT3 is able to follow instructions, but InstructGPT (and ChatGTP which I believe was trained with a similar method to InstructGPT) are better at it.  The ability likely emerged because some of the training data contained examples of people doing it.
Comments.UserDisplayName: occipita
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-01T11:01:46.297Z
Comments.Text: This entry is likely to help answer the question: https://openai.com/blog/instruction-following/
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-02T12:10:22.487Z
Comments.Text: That helps a lot. In other words, it's not truly emergent behaviour, it was specifically trained for.  Maybe I just misunderstood the article. If you can add this as an answer, I will accept it.
Comments.UserDisplayName: ShankarG
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-02T16:31:33.587Z
Comments.Text: From [review](https://ai.stackexchange.com/review/first-questions/24224): this *question* falls on the line of **too many** questions. You should ask only [one question per question](https://meta.stackexchange.com/a/92694/282094). Since these **three** questions are so closely related, and could be answered in one paragraph, I'll not complain. The problem with multiple questions is that the question can be too broad or part of the single answer may be great while others portions fail to address the remaining questions. Please be careful to compose questions that only ask one thing. Thanks.
Comments.UserDisplayName: Rob
Tags: <neural-networks><transformer><gpt>
Title: "Following instructions" as an emergent behaviour in transformer models - isn't this fundamentally different from the models' basic purpose?
UserDisplayName: ShankarG
ThreadText: # "Following instructions" as an emergent behaviour in transformer models - isn't this fundamentally different from the models' basic purpose?
  
  (Asked by: username_0 on 2023-01-31)
  
  I am not technically familiar with AI or neural networks beyond a tech news reading level of knowledge, so I apologise if this is a dumb question.
  
  I was recently reading [this article](https://arstechnica.com/gadgets/2023/01/the-generative-ai-revolution-has-begun-how-did-we-get-here/3/) on [Ars Technica](https://www.arstechnica.com). It is a high level description of the history of generative AI models (it's a very good article, highly recommended).
  
  When discussing large language models, the following passage appears:
  
  > But there was also a surprise. The OpenAI researchers discovered that in making the models bigger, they didn’t just get better at producing text. The models could learn entirely new behaviors simply by being shown new training data. In particular, **the researchers discovered that GPT3 could be trained to follow instructions in plain English without having to explicitly design the model that way.** (emphasis added)
  
  I have three questions:
  
  1. Was this instruction-following behaviour truly emergent, as in completely unexpected and unplanned for?
  2. This seems completely at odds with the usual description of transformer models as transforming text of one kind into another. Following instructions seems, at a conceptual level, to be something much higher? If this is the case, do we have any idea how this emerged, and what properties of the model it is rooted in?
  3. Do we have any idea how wide the scope of this kind of "instruction following" is? I.e. can the models make sense and respond "sensibly" (not correctly - I am not concerned with the correctness of the response, but with its relation to the instruction) to *any* instruction related to text? Or are there specific kinds of instructions they are able to comprehend, and others they fail at?
  
  **Tags:** neural-networks, transformer, gpt
  > **Comments:**
  > **username_2**: This entry is likely to help answer the question: https://openai.com/blog/instruction-following/  
  > **username_0**: That helps a lot. In other words, it's not truly emergent behaviour, it was specifically trained for.  Maybe I just misunderstood the article. If you can add this as an answer, I will accept it.  
  > **username_3**: From [review](https://ai.stackexchange.com/review/first-questions/24224): this *question* falls on the line of **too many** questions. You should ask only [one question per question](https://meta.stackexchange.com/a/92694/282094). Since these **three** questions are so closely related, and could be answered in one paragraph, I'll not complain. The problem with multiple questions is that the question can be too broad or part of the single answer may be great while others portions fail to address the remaining questions. Please be careful to compose questions that only ask one thing. Thanks.  
  > **username_1**: I'm not completely au-fait with the topic, so can't really answer it myself, but I think you misunderstand the link.  "it's not truly emergent behaviour, it was specifically trained for" -- what I take away from the link is that it *was* emergent behaviour, but then OpenAI fine tuned their training methods in order to make it better.  Unoptimized GPT3 is able to follow instructions, but InstructGPT (and ChatGTP which I believe was trained with a similar method to InstructGPT) are better at it.  The ability likely emerged because some of the training data contained examples of people doing it.  
  
  ---
  
  ## 0 answers
  
  