Answers.Body: <p>They say “a pretrained GPT-3” model, emphasis on “a” implying one of many, rather than “the”.</p>
  <p>I believe they simply repeat the process with various parameters scales of the pretrained GPT-3, comparing the performance to GPT-3 175B throughout to see if there are parameter efficiency gains.</p>
  <p>They note that even the 1.3B instructGPT version outperforms the 175B GPT-3 version.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-25T11:37:30.800Z
Answers.LastActivityDate: 2023-02-25T11:37:30.800Z
Answers.UserDisplayName: thesofakillers
Body: <p>I am reading the paper &quot;<a href="https://arxiv.org/pdf/2203.02155" rel="nofollow noreferrer">Training language models to follow instructions with human feedback</a>&quot;</p>
  <p>It says:</p>
  <blockquote>
  <p>Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning.</p>
  </blockquote>
  <p>The paper also says:</p>
  <blockquote>
  <p>On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3,
  despite having over 100x fewer parameters.</p>
  </blockquote>
  <p>I am not able to understand how the aligned model is a fine tune of GPT-3 using supervised learning (and other steps associated with reinforcement learning) and at the same time the aligned model has fewer parameters than the original model.</p>
  <p>Can someone give me a hint on the subject?</p>
  
Tags: <weights><gpt-3><instruct-gpt>
Title: How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?
UserDisplayName: DanielTheRocketMan
ThreadText: # How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?
  
  (Asked by: username_0 on 2023-02-25)
  
  I am reading the paper "[Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155)"
  
  It says:
  
  > Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning.
  
  The paper also says:
  
  > On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3,
  > despite having over 100x fewer parameters.
  
  I am not able to understand how the aligned model is a fine tune of GPT-3 using supervised learning (and other steps associated with reinforcement learning) and at the same time the aligned model has fewer parameters than the original model.
  
  Can someone give me a hint on the subject?
  
  **Tags:** weights, gpt-3, instruct-gpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-25. Score: 2 | Accepted answer
  
  They say “a pretrained GPT-3” model, emphasis on “a” implying one of many, rather than “the”.
  
  I believe they simply repeat the process with various parameters scales of the pretrained GPT-3, comparing the performance to GPT-3 175B throughout to see if there are parameter efficiency gains.
  
  They note that even the 1.3B instructGPT version outperforms the 175B GPT-3 version.
  
  
  ---
  
  