Answers.Body: <p>If you haven't already, I would recommend a careful reading of <a href="https://arxiv.org/pdf/2203.02155.pdf" rel="nofollow noreferrer">OpenAI's paper on InstructGPT</a>. This was their publication from last year regarding how they applied RLHF to GPT-3, the precursor of ChatGPT.</p>
  <p>The appendix provides information on the reward model and the RLHF training data. For example,</p>
  <blockquote>
  <p>For the reward models and
  value functions, the unembedding layer of the original model is replaced with a projection layer
  to output a scalar value.</p>
  </blockquote>
  <blockquote>
  <p>The final reward model was initialized from a 6B GPT-3 model that was fine-tuned on a variety of
  public NLP datasets (ARC, BoolQ, CoQA, DROP, MultiNLI, OpenBookQA, QuAC, RACE, and
  Winogrande).</p>
  </blockquote>
  <p>and,</p>
  <blockquote>
  <p>We train all the RL models for 256k episodes. These episodes include
  about 31k unique prompts, after filtering out prompts with PII and deduplication based on common
  prefixes.</p>
  </blockquote>
  <p>If you want to know what <em>ChatGPT</em> does specifically, you might have to ask someone who works there. It's not public information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T09:33:46.110Z
Answers.Text: There does not have to be anything published given that RLHF is by now done by a ton of open source projects.
Answers.UserDisplayName: TomTom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T17:13:10.387Z
Answers.LastActivityDate: 2023-07-12T17:13:10.387Z
Answers.UserDisplayName: Venna Banana
Body: <p>My rough understanding of RLHF as used for ChatGPT in a nutshell is this:</p>
  <ol>
  <li><p>A reward model is trained using comparisons of different responses
  to the same prompt. Human trainers rank these responses based on
  quality.</p>
  </li>
  <li><p>The reward model is a neural network that learns to predict these
  human rankings. It essentially learns the &quot;policy&quot; that human
  trainers use to rank responses.</p>
  </li>
  <li><p>An initial policy, which is a language model, is fine-tuned using
  Proximal Policy Optimization (PPO) with the reward model providing
  the reward signal. This process is iterative, with the policy and
  reward model being updated alternately.</p>
  </li>
  <li><p>The policy is then used to generate responses to prompts. The reward
  model assesses these responses and provides a reward signal, which
  is used to further fine-tune the policy, i.e. the language model.</p>
  </li>
  </ol>
  <p>My main question is the first one, the others are just for giving context:</p>
  <blockquote>
  <p><strong>1. What's the architecture and size of the neural-network-based reward model?</strong></p>
  <ol start="2">
  <li><p>Is it pretrained, too? Is it possibly another pretrained (foundational) language model?</p>
  </li>
  <li><p>By how many samples labelled by human trainers is the reward model trained?</p>
  </li>
  <li><p>By how many prompts and rewarded completions is the language model trained later? (Which prompts, by the way?)</p>
  </li>
  </ol>
  </blockquote>
  <p>These numbers I'd like to compare with the numbers of pretrained ChatGPT:</p>
  <ul>
  <li><p>Transformer-based ChatGPT has 175 billion weights.</p>
  </li>
  <li><p>It was pretrained on 500 GB of text data, distributed over an unknown number of &quot;documents&quot; (from single tweets to the Holy Bible) with roughly 500B tokens over all. During training ChatGPT was exposed to a multiple of 500B samples (assuming that all 500B tokens were used for training).</p>
  </li>
  </ul>
  <p>I assume that during RLHF foundational ChatGPT was exposed to a much smaller number of prompts to complete (and to be rewarded).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-09T23:14:34.660Z
Comments.Text: It seems you're asking many questions here, which makes this post too broad. If you're asking many questions, please, pick one, then ask the others in separate posts, unless they are really sub-questions that would answer the main question. Once you've picked your main question, please, put it in the title for clarity.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-10T09:47:35.060Z
Comments.Text: @nbro: Is it better now?
Comments.UserDisplayName: Hans-Peter Stricker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-11T07:53:37.513Z
Comments.Text: @nbro: Can you imagine why this question is not so well received: viewed only rarely and even downvoted? Is it ill-posed or trivial?
Comments.UserDisplayName: Hans-Peter Stricker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-12T12:31:43.873Z
Comments.Text: There has been a lot of hype around ChatGPT, LLM, etc. As you probably have noticed, hype comes and goes. I think your questions are interesting, more interesting than those like  "Why can't ChatGPT do math?", or stuff like that, which are boring to me. You're asking technical/theoretical questions, which seems to be suitable for our site. Last time I checked, there was no official ChatGPT paper. The only official details that we have about ChatGPT are only in the OpenAI posts on the topic. So, maybe this info is not yet available. I'd need to check there. The InstructGPT paper may be useful
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-12T12:41:00.867Z
Comments.Text: Anyway, maybe your post can still be simplified. To me, it still seems you're asking different questions about the reward model. I don't know if removing all but 1 question will help to attract more people. To me, your post is still asking the question: "Can you please provide more details about the reward model used in ChatGPT?" and then you ask some questions with specific details you're interested in. Maybe it's fine to keep it like that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-12T22:14:20.490Z
Comments.Text: AFAIK, the GPT transformer is asked to output an additional token at the end of the sequence it has generated with the expected score (it's fine tuned to learn this), thus the reward network is also the language model... you can find a lot of insights in this video https://www.youtube.com/watch?v=bZQun8Y4L2A&t=2254s&ab_channel=MicrosoftDeveloper from microsoft
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning><rewards><hyper-parameters><training-datasets><human-like>
Title: What's the architecture and size of neural-network-based reward models as used in reinforcement learning by human feedback
UserDisplayName: Hans-Peter Stricker
ThreadText: # What's the architecture and size of neural-network-based reward models as used in reinforcement learning by human feedback
  
  (Asked by: username_0 on 2023-07-06)
  
  My rough understanding of RLHF as used for ChatGPT in a nutshell is this:
  
  1. A reward model is trained using comparisons of different responses
     to the same prompt. Human trainers rank these responses based on
     quality.
  2. The reward model is a neural network that learns to predict these
     human rankings. It essentially learns the "policy" that human
     trainers use to rank responses.
  3. An initial policy, which is a language model, is fine-tuned using
     Proximal Policy Optimization (PPO) with the reward model providing
     the reward signal. This process is iterative, with the policy and
     reward model being updated alternately.
  4. The policy is then used to generate responses to prompts. The reward
     model assesses these responses and provides a reward signal, which
     is used to further fine-tune the policy, i.e. the language model.
  
  My main question is the first one, the others are just for giving context:
  
  > **1. What's the architecture and size of the neural-network-based reward model?**
  >
  > 2. Is it pretrained, too? Is it possibly another pretrained (foundational) language model?
  > 3. By how many samples labelled by human trainers is the reward model trained?
  > 4. By how many prompts and rewarded completions is the language model trained later? (Which prompts, by the way?)
  
  These numbers I'd like to compare with the numbers of pretrained ChatGPT:
  
  * Transformer-based ChatGPT has 175 billion weights.
  * It was pretrained on 500 GB of text data, distributed over an unknown number of "documents" (from single tweets to the Holy Bible) with roughly 500B tokens over all. During training ChatGPT was exposed to a multiple of 500B samples (assuming that all 500B tokens were used for training).
  
  I assume that during RLHF foundational ChatGPT was exposed to a much smaller number of prompts to complete (and to be rewarded).
  
  **Tags:** reinforcement-learning, rewards, hyper-parameters, training-datasets, human-like
  
  > **Comments:**
  > **username_1**: It seems you're asking many questions here, which makes this post too broad. If you're asking many questions, please, pick one, then ask the others in separate posts, unless they are really sub-questions that would answer the main question. Once you've picked your main question, please, put it in the title for clarity.  
  > **username_0**: @username_1: Is it better now?  
  > **username_0**: @username_1: Can you imagine why this question is not so well received: viewed only rarely and even downvoted? Is it ill-posed or trivial?  
  > **username_1**: There has been a lot of hype around ChatGPT, LLM, etc. As you probably have noticed, hype comes and goes. I think your questions are interesting, more interesting than those like  "Why can't ChatGPT do math?", or stuff like that, which are boring to me. You're asking technical/theoretical questions, which seems to be suitable for our site. Last time I checked, there was no official ChatGPT paper. The only official details that we have about ChatGPT are only in the OpenAI posts on the topic. So, maybe this info is not yet available. I'd need to check there. The InstructGPT paper may be useful  
  > **username_1**: Anyway, maybe your post can still be simplified. To me, it still seems you're asking different questions about the reward model. I don't know if removing all but 1 question will help to attract more people. To me, your post is still asking the question: "Can you please provide more details about the reward model used in ChatGPT?" and then you ask some questions with specific details you're interested in. Maybe it's fine to keep it like that.  
  > **username_2**: AFAIK, the GPT transformer is asked to output an additional token at the end of the sequence it has generated with the expected score (it's fine tuned to learn this), thus the reward network is also the language model... you can find a lot of insights in this video https://www.youtube.com/watch?v=bZQun8Y4L2A&t=2254s&ab_channel=MicrosoftDeveloper from microsoft  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-07-12. Score: 1 | Accepted answer
  
  If you haven't already, I would recommend a careful reading of [OpenAI's paper on InstructGPT](https://arxiv.org/pdf/2203.02155.pdf). This was their publication from last year regarding how they applied RLHF to GPT-3, the precursor of ChatGPT.
  
  The appendix provides information on the reward model and the RLHF training data. For example,
  
  > For the reward models and
  > value functions, the unembedding layer of the original model is replaced with a projection layer
  > to output a scalar value.
  
  > The final reward model was initialized from a 6B GPT-3 model that was fine-tuned on a variety of
  > public NLP datasets (ARC, BoolQ, CoQA, DROP, MultiNLI, OpenBookQA, QuAC, RACE, and
  > Winogrande).
  
  and,
  
  > We train all the RL models for 256k episodes. These episodes include
  > about 31k unique prompts, after filtering out prompts with PII and deduplication based on common
  > prefixes.
  
  If you want to know what *ChatGPT* does specifically, you might have to ask someone who works there. It's not public information.
  
  > **Comments:**
  > **username_4**: There does not have to be anything published given that RLHF is by now done by a ton of open source projects.  
  
  ---
  
  