Answers.Body: <p>About the first point, is most likely that the network will receive the full completion as you say, in order to estimate the reward (though, you can definitely just feed just <span class="math-container">$[x, next\, token]$</span>, however the reward model will have a hard time giving a meaningful reward)</p>
  <p>Regarding the second part, yes, <span class="math-container">$r_\theta(x,y)$</span> is constant, however the expectation is over <span class="math-container">$(x,y)$</span> pairs coming from <span class="math-container">$\pi_\phi^{RL}$</span>, which changes over time, therefore the model has find a way to maximize the reward model, without moving too far from the SFT model</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-13T14:59:50.370Z
Answers.LastActivityDate: 2024-03-13T14:59:50.370Z
Answers.UserDisplayName: Alberto
Body: <p>In the <a href="https://arxiv.org/pdf/2203.02155.pdf" rel="nofollow noreferrer">paper</a>,</p>
  <p>They write:</p>
  <p><a href="https://i.sstatic.net/DZjlz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DZjlz.png" alt="enter image description here" /></a></p>
  <p>Now, is <span class="math-container">$y$</span> the full response or only the next token repsonse? One the one hand, the reward model expects full response, on the other hand they write 'per-token KL penalty'. So we sample the next token, or we sample the next full response?</p>
  <p>Secondly, is the equation correct? After all it looks they only optimize for lower KL divergence, but where's the main optimization taken place? (The reward model is fixed in this equation)</p>
  
Tags: <reinforcement-learning><proximal-policy-optimization><instruct-gpt>
Title: InstructGPT: In the objective RL function, does $y$ mean full response, or only a single token?
UserDisplayName: Nathan G
ThreadText: # InstructGPT: In the objective RL function, does $y$ mean full response, or only a single token?
  
  (Asked by: username_0 on 2024-03-13)
  
  In the [paper](https://arxiv.org/pdf/2203.02155.pdf),
  
  They write:
  
  [![enter image description here](https://i.sstatic.net/DZjlz.png)](https://i.sstatic.net/DZjlz.png)
  
  Now, is $y$ the full response or only the next token repsonse? One the one hand, the reward model expects full response, on the other hand they write 'per-token KL penalty'. So we sample the next token, or we sample the next full response?
  
  Secondly, is the equation correct? After all it looks they only optimize for lower KL divergence, but where's the main optimization taken place? (The reward model is fixed in this equation)
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, instruct-gpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-13. Score: 1
  
  About the first point, is most likely that the network will receive the full completion as you say, in order to estimate the reward (though, you can definitely just feed just $[x, next\, token]$, however the reward model will have a hard time giving a meaningful reward)
  
  Regarding the second part, yes, $r\_\theta(x,y)$ is constant, however the expectation is over $(x,y)$ pairs coming from $\pi\_\phi^{RL}$, which changes over time, therefore the model has find a way to maximize the reward model, without moving too far from the SFT model
  
  
  ---
  
  