Answers.Body: <p>Recall that RLHF essentially has three stages:</p>
  <ul>
  <li>Step 0: Pretrain a text-generating LLM.</li>
  <li>Step 1: Collect human ratings on text, and from that, learn a reward model <span class="math-container">$r_{\theta}$</span> which can provide ratings on any unseen text. This model may use parts of the LLM. (<em>This is where you backpropagate through the reward model</em>!)</li>
  <li>Step 2: Finetune the parameters <span class="math-container">$\Phi$</span> of your text-generating model, <span class="math-container">$LLM_{\Phi}$</span> with respect to a <strong>frozen</strong> reward function <span class="math-container">$r_{\theta}$</span> to maximize the rewards.</li>
  </ul>
  <h3>Q1: In Step 2, why don't we apply gradients through the (frozen) reward function?</h3>
  <p>It seems there is some confusion around what it means to <em>maximize the rewards</em>. In RL, you think of the the reward function <span class="math-container">$r_\theta$</span> as something dictated by the environment --- you can't change it, <em>nor does it make sense to change the reward itself</em> however differentiable it may be. What does make sense is to take actions with our agent's policy, and see how the environment rewards them, and change the agent's policy to &quot;maximize the rewards&quot;.</p>
  <p>Analogously, the idea in RLHF is to use a (fixed) reward function <span class="math-container">$r_{\theta}(y)$</span> which can provide useful ratings of text <span class="math-container">$y$</span> generated from an <span class="math-container">$LLM_{\Phi}$</span>. This can be used to improve the parameters <span class="math-container">$\phi$</span> of <span class="math-container">$LLM_{\Phi}$</span> into assigning more likelihood to text that are of higher quality.  Concretely Step 2 involves:</p>
  <ul>
  <li>(a) making your agent take various independent action(s) according to your current policy i.e., make <span class="math-container">$LLM_{\Phi}$</span> generate various text completions, <span class="math-container">$y_1, y_2, ...$</span> given a prompt <span class="math-container">$p$</span></li>
  <li>(b) then looking at the reward(s), <span class="math-container">$r_\theta(y_i)$</span></li>
  <li>(c) then adjusting <span class="math-container">$\Phi$</span> (NOT <span class="math-container">$\theta$</span>) to increase/decrease the probability of that (series of) action(s) according to the reward.</li>
  </ul>
  <p>Hope you see why it does not make sense to differentiate through the reward model itself. You want to use the useful signal from the reward model to improve your generator.</p>
  <h3>Q2: Why do we need a neural network for the reward function in the first place?</h3>
  <p>Ideally, a human should provide feedback on all the generated text. That would be infeasible. So we resort to training a reward model that learns from human feedback, and generalizes its ratings to unseen text. Indeed, at this stage of RLHF, we differentiate through <span class="math-container">$r_{\theta}$</span>. But after that <span class="math-container">$\theta$</span> is frozen.</p>
  <p>You may still ask:</p>
  <h3>Q3: Given that we have <span class="math-container">$r_{\theta}(x)$</span> where <span class="math-container">$x$</span> is the output of <span class="math-container">$LLM_{\Phi}$</span>, why not pass gradients through <span class="math-container">$r_{\theta}(x)$</span> to update <span class="math-container">$\Phi$</span>?</h3>
  <p>This is difficult to do because <span class="math-container">$x$</span> is a discrete variable. Instead what we do is to change the distribution over <span class="math-container">$x$</span> so as to maximize <span class="math-container">$r_{\theta}(x)$</span>. This does not require &quot;backpropagating through&quot; <span class="math-container">$r_{\theta}(x)$</span>. Instead, this only requires updating the likelihood of <span class="math-container">$x$</span> under <span class="math-container">$LLM_{\Phi}$</span>, proportional to <span class="math-container">$r_{\theta}(x)$</span>.</p>
  <hr />
  <p><strong>Note:</strong> The rest is my older answer, which may not have addressed your original question.</p>
  <p>You may still ask, within Step (c) above:</p>
  <h3>Q4. Why PPO? Why not simple SGD to update the likelihood?</h3>
  <p>(but I realize this may not be your question?)</p>
  <p><strong>PPO <em>is</em> in essence doing a simple SGD, but with some fancy penalties for the sake of stability.</strong> The &quot;simple SGD&quot; part of PPO takes care of updating your &quot;policy&quot; so as to maximize your &quot;reward&quot;. This update on <span class="math-container">$\Phi$</span> (NOT on <span class="math-container">$\theta$</span>), improves the likelihood of <span class="math-container">$LLM_{\Phi}$</span> on text that <span class="math-container">$r_{\theta}$</span> assigns higher rating to.</p>
  <p>Meanwhile, the &quot;fancy penalty&quot; makes sure that in doing your simple update, your policy remains in sufficient proximity to your &quot;original policy&quot;. Hence the name, Proximal Policy Optimization. Without this penalty, the agent may diverge out of instability and start producing garbage as explained in the next question. See <a href="https://huggingface.co/blog/deep-rl-ppo" rel="nofollow noreferrer">https://huggingface.co/blog/deep-rl-ppo</a> for a nice technical discussion of the algorithm.</p>
  <h3>Q5. What exactly are these instabilities?</h3>
  <p>Recall that what you have is a reward model trained on a specific kind of data (e.g., meaningful text from a pre-trained LLM). Thus, it is possible that your reward model has no clue how to score text from other distributions: as an extreme case, imagine complete gibberish text. In fact, it's even possible that the global maximum with respect to the learned reward model, lurks in some gibberish/strange space of language where the reward model wasn't trained on. If your LLMs by happenstance produces a gibberish that the reward model thinks is great, the LLM may quickly devolve into fooling your reward model by producing only said gibberish.</p>
  <h3>Q6. Can't you simulate this with a small learning rate with simple SGD?</h3>
  <p>I do not have a good answer to this as I'm not an RL expert. My guess would be either that (i) this may require limiting to very small learning rate that makes learning too slow or impossible (perhaps precluding any type of feature-learning) or (ii) it may still be possible to get caught in bad local maxima, since this corresponds to purely maximizing only the reward objective.</p>
  <p>Now, going by the title of your question, perhaps you may still ask (or maybe you don't; I did wonder about this when I first came across RLHF for LLMs):</p>
  <h3>Q7. Isn't this just SGD with some supervision and a fancy penalty? Why is this called RL?</h3>
  <p>Remember that in supervised learning, you're given an immutable set of labeled input-output pairs, while in RL, you're <em>interacting</em> with your environment by producing your own inputs and <em>collect</em> output values in response to that. Analogously, in standard LLM training, you are given an immutable set of human-generated text for supervision on which you maximize likelihood. In RLHF however, the LLM has the freedom to generate whatever text it may please as in step (a), and gather supervision on that text, as in step (b). This is exactly why it can generate &quot;whatever it may please&quot; and devolve into a gibberish state (which in turn motivates a need to fetter your policy close to the initial policy).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T17:02:27.533Z
Answers.Text: Thank you for the answer. Unfortunately I won't be able to go through it before the weekend, so please bear with me.
Answers.UserDisplayName: DeltaIV
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T21:46:42.417Z
Answers.Text: No problem. I had to update your answer because I realize you probably had confusion with something else. I left my original answer as it is for the sake of posterity.
Answers.UserDisplayName: Enigman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-24T18:20:50.340Z
Answers.LastActivityDate: 2023-04-27T00:04:13.227Z
Answers.LastEditDate: 2023-04-27T00:04:13.227Z
Answers.UserDisplayName: Enigman
Body: <p>In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see <a href="https://arxiv.org/abs/2203.02155" rel="nofollow noreferrer">https://arxiv.org/abs/2203.02155</a>). Then, why do we need PPO to find that policy? Since the reward function is differentiable and cheap to evaluate, wouldn't some gradient descent algorithm such as SGD or Adam be enough?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-24T21:27:45.163Z
Comments.Text: I don't understand this question. It seems you don't understand the difference between a reward function and policy. A policy tells you how to act. A reward function does **not**. That should answer your question why we want to learn a policy and just don't stick with a reward function. I am even more surprised by this question given that you answered a question on RL here: https://ai.stackexchange.com/a/13123/2444 (that I asked a few years ago). Maybe I'm just misunderstanding what you're asking here. In that case, maybe your post should be clarified.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-26T17:08:19.430Z
Comments.Text: @nbro of course I never asked "why we need to learn a policy". Where did you get that from? I asked why we use PPO to learn the policy instead of SGD, given that the reward is differentiable and thus we could easily compute/estimate its gradient. Enigman below got the meaning of my question right.
Comments.UserDisplayName: DeltaIV
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-26T17:43:12.607Z
Comments.Text: Let's assume the reward function is differentiable. Then you would use SGD for what? How? You don't explain how you would do that. So, your question is unclear, yes. It's unclear if you have any knowledge of PPO or RL at all from your post. Maybe they got your "question" right, but they had to actually write a essay and answer apparently 4 questions, not just one. So, how come? The answer to your question "why do we need PPO to find that policy?" is simple - we don't strictly need it, but we use it because that's an RL algorithm that works.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-26T17:50:40.250Z
Comments.Text: Let's define the differentiable reward function as $r(s, a)$. Then you would differentiate it with respect to what? $a$? $s$? To obtain which minimum/maximum? Do we really want to find a minimum/maximum of the reward function? It seems that's your understanding - we just want to optimize the reward function. But that's not the case. You want to find a policy, so find the parameters of a policy. But what's the objective function here? It's not the reward function, actually. That's why I think this unclear. It seems you're mixing the concepts of objective function, reward function and policy.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-26T17:55:11.443Z
Comments.Text: But there are so many things that I'm trying to assume or hypothesize here. So, that should be a clear sign that your post is unclear, but ok. If they guessed your doubts and satisfactorily answer your questions, great.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><chatgpt><large-language-models><rlhf>
Title: Why do we need RL in RLHF?
UserDisplayName: DeltaIV
ThreadText: # Why do we need RL in RLHF?
  
  (Asked by: username_0 on 2023-03-02)
  
  In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see <https://arxiv.org/abs/2203.02155>). Then, why do we need PPO to find that policy? Since the reward function is differentiable and cheap to evaluate, wouldn't some gradient descent algorithm such as SGD or Adam be enough?
  
  **Tags:** reinforcement-learning, chatgpt, large-language-models, rlhf
  
  > **Comments:**
  > **username_1**: I don't understand this question. It seems you don't understand the difference between a reward function and policy. A policy tells you how to act. A reward function does **not**. That should answer your question why we want to learn a policy and just don't stick with a reward function. I am even more surprised by this question given that you answered a question on RL here: https://ai.stackexchange.com/a/13123/2444 (that I asked a few years ago). Maybe I'm just misunderstanding what you're asking here. In that case, maybe your post should be clarified.  
  > **username_0**: @username_1 of course I never asked "why we need to learn a policy". Where did you get that from? I asked why we use PPO to learn the policy instead of SGD, given that the reward is differentiable and thus we could easily compute/estimate its gradient. username_2 below got the meaning of my question right.  
  > **username_1**: Let's assume the reward function is differentiable. Then you would use SGD for what? How? You don't explain how you would do that. So, your question is unclear, yes. It's unclear if you have any knowledge of PPO or RL at all from your post. Maybe they got your "question" right, but they had to actually write a essay and answer apparently 4 questions, not just one. So, how come? The answer to your question "why do we need PPO to find that policy?" is simple - we don't strictly need it, but we use it because that's an RL algorithm that works.  
  > **username_1**: Let's define the differentiable reward function as $r(s, a)$. Then you would differentiate it with respect to what? $a$? $s$? To obtain which minimum/maximum? Do we really want to find a minimum/maximum of the reward function? It seems that's your understanding - we just want to optimize the reward function. But that's not the case. You want to find a policy, so find the parameters of a policy. But what's the objective function here? It's not the reward function, actually. That's why I think this unclear. It seems you're mixing the concepts of objective function, reward function and policy.  
  > **username_1**: But there are so many things that I'm trying to assume or hypothesize here. So, that should be a clear sign that your post is unclear, but ok. If they guessed your doubts and satisfactorily answer your questions, great.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-24. Score: 4
  
  Recall that RLHF essentially has three stages:
  
  * Step 0: Pretrain a text-generating LLM.
  * Step 1: Collect human ratings on text, and from that, learn a reward model $r\_{\theta}$ which can provide ratings on any unseen text. This model may use parts of the LLM. (*This is where you backpropagate through the reward model*!)
  * Step 2: Finetune the parameters $\Phi$ of your text-generating model, $LLM\_{\Phi}$ with respect to a **frozen** reward function $r\_{\theta}$ to maximize the rewards.
  
  ### Q1: In Step 2, why don't we apply gradients through the (frozen) reward function?
  
  It seems there is some confusion around what it means to *maximize the rewards*. In RL, you think of the the reward function $r\_\theta$ as something dictated by the environment --- you can't change it, *nor does it make sense to change the reward itself* however differentiable it may be. What does make sense is to take actions with our agent's policy, and see how the environment rewards them, and change the agent's policy to "maximize the rewards".
  
  Analogously, the idea in RLHF is to use a (fixed) reward function $r\_{\theta}(y)$ which can provide useful ratings of text $y$ generated from an $LLM\_{\Phi}$. This can be used to improve the parameters $\phi$ of $LLM\_{\Phi}$ into assigning more likelihood to text that are of higher quality. Concretely Step 2 involves:
  
  * (a) making your agent take various independent action(s) according to your current policy i.e., make $LLM\_{\Phi}$ generate various text completions, $y\_1, y\_2, ...$ given a prompt $p$
  * (b) then looking at the reward(s), $r\_\theta(y\_i)$
  * (c) then adjusting $\Phi$ (NOT $\theta$) to increase/decrease the probability of that (series of) action(s) according to the reward.
  
  Hope you see why it does not make sense to differentiate through the reward model itself. You want to use the useful signal from the reward model to improve your generator.
  
  ### Q2: Why do we need a neural network for the reward function in the first place?
  
  Ideally, a human should provide feedback on all the generated text. That would be infeasible. So we resort to training a reward model that learns from human feedback, and generalizes its ratings to unseen text. Indeed, at this stage of RLHF, we differentiate through $r\_{\theta}$. But after that $\theta$ is frozen.
  
  You may still ask:
  
  ### Q3: Given that we have $r\_{\theta}(x)$ where $x$ is the output of $LLM\_{\Phi}$, why not pass gradients through $r\_{\theta}(x)$ to update $\Phi$?
  
  This is difficult to do because $x$ is a discrete variable. Instead what we do is to change the distribution over $x$ so as to maximize $r\_{\theta}(x)$. This does not require "backpropagating through" $r\_{\theta}(x)$. Instead, this only requires updating the likelihood of $x$ under $LLM\_{\Phi}$, proportional to $r\_{\theta}(x)$.
  
  ---
  
  **Note:** The rest is my older answer, which may not have addressed your original question.
  
  You may still ask, within Step (c) above:
  
  ### Q4. Why PPO? Why not simple SGD to update the likelihood?
  
  (but I realize this may not be your question?)
  
  **PPO *is* in essence doing a simple SGD, but with some fancy penalties for the sake of stability.** The "simple SGD" part of PPO takes care of updating your "policy" so as to maximize your "reward". This update on $\Phi$ (NOT on $\theta$), improves the likelihood of $LLM\_{\Phi}$ on text that $r\_{\theta}$ assigns higher rating to.
  
  Meanwhile, the "fancy penalty" makes sure that in doing your simple update, your policy remains in sufficient proximity to your "original policy". Hence the name, Proximal Policy Optimization. Without this penalty, the agent may diverge out of instability and start producing garbage as explained in the next question. See <https://huggingface.co/blog/deep-rl-ppo> for a nice technical discussion of the algorithm.
  
  ### Q5. What exactly are these instabilities?
  
  Recall that what you have is a reward model trained on a specific kind of data (e.g., meaningful text from a pre-trained LLM). Thus, it is possible that your reward model has no clue how to score text from other distributions: as an extreme case, imagine complete gibberish text. In fact, it's even possible that the global maximum with respect to the learned reward model, lurks in some gibberish/strange space of language where the reward model wasn't trained on. If your LLMs by happenstance produces a gibberish that the reward model thinks is great, the LLM may quickly devolve into fooling your reward model by producing only said gibberish.
  
  ### Q6. Can't you simulate this with a small learning rate with simple SGD?
  
  I do not have a good answer to this as I'm not an RL expert. My guess would be either that (i) this may require limiting to very small learning rate that makes learning too slow or impossible (perhaps precluding any type of feature-learning) or (ii) it may still be possible to get caught in bad local maxima, since this corresponds to purely maximizing only the reward objective.
  
  Now, going by the title of your question, perhaps you may still ask (or maybe you don't; I did wonder about this when I first came across RLHF for LLMs):
  
  ### Q7. Isn't this just SGD with some supervision and a fancy penalty? Why is this called RL?
  
  Remember that in supervised learning, you're given an immutable set of labeled input-output pairs, while in RL, you're *interacting* with your environment by producing your own inputs and *collect* output values in response to that. Analogously, in standard LLM training, you are given an immutable set of human-generated text for supervision on which you maximize likelihood. In RLHF however, the LLM has the freedom to generate whatever text it may please as in step (a), and gather supervision on that text, as in step (b). This is exactly why it can generate "whatever it may please" and devolve into a gibberish state (which in turn motivates a need to fetter your policy close to the initial policy).
  
  > **Comments:**
  > **username_0**: Thank you for the answer. Unfortunately I won't be able to go through it before the weekend, so please bear with me.  
  > **username_2**: No problem. I had to update your answer because I realize you probably had confusion with something else. I left my original answer as it is for the sake of posterity.  
  
  ---
  
  