<p>They say “a pretrained GPT-3” model, emphasis on “a” implying one of many, rather than “the”.</p>
<p>I believe they simply repeat the process with various parameters scales of the pretrained GPT-3, comparing the performance to GPT-3 175B throughout to see if there are parameter efficiency gains.</p>
<p>They note that even the 1.3B instructGPT version outperforms the 175B GPT-3 version.</p>

CC BY-SA 4.0
2023-02-25T11:37:30.800Z
2023-02-25T11:37:30.800Z
thesofakillers
<p>I am reading the paper &quot;<a href="https://arxiv.org/pdf/2203.02155" rel="nofollow noreferrer">Training language models to follow instructions with human feedback</a>&quot;</p>
<p>It says:</p>
<blockquote>
<p>Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning.</p>
</blockquote>
<p>The paper also says:</p>
<blockquote>
<p>On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3,
despite having over 100x fewer parameters.</p>
</blockquote>
<p>I am not able to understand how the aligned model is a fine tune of GPT-3 using supervised learning (and other steps associated with reinforcement learning) and at the same time the aligned model has fewer parameters than the original model.</p>
<p>Can someone give me a hint on the subject?</p>

<weights><gpt-3><instruct-gpt>
How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?
DanielTheRocketMan
# How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?

(Asked by: username_0 on 2023-02-25)

I am reading the paper "[Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155)"

It says:

> Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning.

The paper also says:

> On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3,
> despite having over 100x fewer parameters.

I am not able to understand how the aligned model is a fine tune of GPT-3 using supervised learning (and other steps associated with reinforcement learning) and at the same time the aligned model has fewer parameters than the original model.

Can someone give me a hint on the subject?

**Tags:** weights, gpt-3, instruct-gpt


---

## 1 answers

### Answer by username_1 on 2023-02-25. Score: 2 | Accepted answer

They say “a pretrained GPT-3” model, emphasis on “a” implying one of many, rather than “the”.

I believe they simply repeat the process with various parameters scales of the pretrained GPT-3, comparing the performance to GPT-3 175B throughout to see if there are parameter efficiency gains.

They note that even the 1.3B instructGPT version outperforms the 175B GPT-3 version.


---

