Answers.Body: <p>According to <a href="https://wandb.ai/ayush-thakur/RLHF/reports/Understanding-Reinforcement-Learning-from-Human-Feedback-RLHF-Part-1--VmlldzoyODk5MTIx" rel="nofollow noreferrer">this guide</a>, the sigma in this formula refers to the sigmoid activation function. The guide does not tell exactly why the sigmoid function is used here, so I will try to give a full explanation of how this loss formulation works (page 8, formula 1 <a href="https://arxiv.org/abs/2203.02155" rel="nofollow noreferrer">in the InstructGPT paper</a>):</p>
  <p><span class="math-container">$\text{loss}(\theta)=-\frac{1}{\binom{K}{2}}E_{(x,y_w,y_l) \sim D} [log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))]$</span></p>
  <p>In the following I will use the notation from the paper:</p>
  <p><span class="math-container">$x$</span> refers to the given instruction</p>
  <p>(<span class="math-container">$y_w$</span>, <span class="math-container">$y_l$</span>) refers to a pair of responses out of the list of responses which a human ranked based on their preference</p>
  <p><span class="math-container">$y_w$</span> refers to the response that is preferred over the other, lesser preferred response <span class="math-container">$y_l$</span></p>
  <p><span class="math-container">$r$</span> refers to the reward model</p>
  <p><span class="math-container">$\theta$</span> refers to the trainable parameters of that reward model</p>
  <p><span class="math-container">$\sigma$</span> refers to the sigmoid activation function.</p>
  <p>If you interpret <span class="math-container">$\sigma(r_\theta(x, y_w) - r_\theta(x, y_l))$</span> as <em>the probability that the reward model assigns a higher reward to the preferred response <span class="math-container">$y_w$</span> than to the lesser preferred response <span class="math-container">$y_l$</span></em>, the formula makes total sense. How do you maximize a probability? Exactly, by minimizing the negative log of that probability (this is called <em>negative log likelihood</em>). Hence the  minus sign in front of the formula and log function around the sigmoid.</p>
  <p>In order to elaborate a bit more on that: If the reward model is working fine, then <span class="math-container">$r_\theta(x, y_w)$</span> will be a very large positive number and <span class="math-container">$r_\theta(x, y_l)$</span> will be a much lower number (maybe even a negative number). The difference <span class="math-container">$r_\theta(x, y_w) - r_\theta(x, y_l)$</span> will then be a very large positive number. And the sigmoid of a very large positive number approaches <span class="math-container">$1$</span>. In that case, everything is according to plan and the loss will be very small (close to zero). However, if the reward model is failing, the assigned reward <span class="math-container">$r_\theta(x, y_w)$</span> might be much smaller than <span class="math-container">$r_\theta(x, y_l)$</span>. Hence, the difference <span class="math-container">$r_\theta(x, y_w) - r_\theta(x, y_l)$</span> will be a (possibly very large) negative number. Take the sigmoid of that and you get a value that approaches <span class="math-container">$0$</span> (thus, the <em>probability that the reward model assigns a higher reward to the preferred response</em> will be small). As we are trying to maximize a probability by minimizing the negative log likelihood, we get a large loss in that case.</p>
  <p>As there will be a varying number of ranked responses for each instruction in one batch (&quot;between <span class="math-container">$K = 4$</span> and
  <span class="math-container">$K = 9$</span> responses&quot;), the losses of those pairwise comparisons must be weighted, so that each instruction has the same impact on the gradient update, no matter how many responses the humans have been presented for each instruction. The number of pairwise comparisons out of <span class="math-container">$K$</span> is <span class="math-container">$\binom{K}{2}$</span>.</p>
  <p>In order to wrap it up: By minimizing the loss described in the paper, the reward model gets incentivized to assign a large positive reward to responses the (hopefully adequately paid <em>cough</em>) humans in front of their computers consider to be very good responses and very large negative rewards to responses which those humans consider to be very bad. And this is exactly what is desired in order to fine-tune an LLM according to human preference using reinforcement learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-07T03:42:00.863Z
Answers.Text: @schreon Thank you for your response. I'm struggling to understand the quantity inside the sigmoid function: (,)−(,). If I am comparing two responses, A and B, and want to calculate the loss of B, if B is ranked higher than A, how is (,)−(,) calculated?
Answers.UserDisplayName: itisyeetimetoday
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-07T19:15:08.850Z
Answers.Text: @itisyeetimetoday first, the loss is based on a comparison of A and B, so asking for only calculating the "loss of B" does not make much sense. You would rather say "the loss of the comparison of A and B, where B should be higher than A".  So if the human annotator said that (given the instruction $x$) the response B is preferred over A, then you assign $y_w = B$ and $y_l = A$. Then you feed $(x, y_w)$ into your reward model $r_\theta$, resulting in one scalar value (the value of the output neuron of your reward model). Repeat this for $(x, y_l)$ and you can calculate that quantity :-)
Answers.UserDisplayName: schreon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-29T11:27:38.647Z
Answers.Text: Thank you. Any idea why it is actually needed here?
Answers.UserDisplayName: Nathan G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T17:22:26.170Z
Answers.Text: To improve this answer could you summarize the specific information from the source that answers the question. This link could become invalid in the future. So by extracting the content that answers the question makes it a stronger and more durable answer for those in the future.
Answers.UserDisplayName: respectful
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-01T08:44:03.483Z
Answers.Text: NathanB and @respectful: alright, I expanded the answer in order to give a full explanation of the loss formula. This is based on the linked guide, the original InstructGPT paper, as well as my personal understanding/interpretation of the material.
Answers.UserDisplayName: schreon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T08:55:56.437Z
Answers.LastActivityDate: 2023-02-01T08:51:53.923Z
Answers.LastEditDate: 2023-02-01T08:51:53.923Z
Answers.UserDisplayName: schreon
Answers.Body: <p>The probability of preferring one trajectory/response (say <span class="math-container">$A$</span>) over the other (say <span class="math-container">$B$</span>) is given by,</p>
  <p><span class="math-container">$$
  \begin{aligned} 
  P(A&gt;B) &amp;= \frac{\exp(r_\theta(A))}{\exp(r_\theta(A))+\exp(r_\theta(B))}\\
  &amp;= \frac{1}{1+\exp(-(r_\theta(A)-r_\theta(B)))}\\
  &amp;= \sigma(r_\theta(A)-r_\theta(B))
  \end{aligned}
  $$</span></p>
  <p>where <span class="math-container">$\sigma(x) = \frac{1}{1+e^{-x}}$</span> is the sigmoid function. This follows from the <a href="https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model" rel="nofollow noreferrer">Bradley-Terry model</a> for estimating score functions from pairwise preferences. If say <span class="math-container">$y_w$</span> and <span class="math-container">$y_l$</span> are the responses from the model (augmented with the query <span class="math-container">$x$</span>), and we prefer <span class="math-container">$y_w$</span> over <span class="math-container">$y_l$</span>, then <span class="math-container">$\mu(y_w)=1; \mu(y_l)=0$</span>. We can use the following cross-entropy loss function and substitute for the probability from above equation to train our reward model:</p>
  <p><span class="math-container">$$
  \begin{aligned}
  L &amp;= -\mu(y_w) \log(P(y_w&gt;y_l)) + \mu(y_l) \log(P(y_l&gt;y_w))\\
  &amp;= -\log(P(y_w&gt;y_l))\\
  &amp;= -\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))
  \end{aligned}
  $$</span></p>
  <p>In InstructGPT, the model is made to generate <span class="math-container">$K$</span> responses. So we can have <span class="math-container">$K\choose2$</span> pairs of comparisons that we can make. Example if the model generates four responses, <span class="math-container">$A, B, C, D$</span> and our ranking is <span class="math-container">$B&gt;C&gt;D&gt;A$</span>, then there are <span class="math-container">${4\choose2}=6$</span> comparisons possible: <span class="math-container">$B&gt;C$</span>, <span class="math-container">$B&gt;D$</span>, <span class="math-container">$B&gt;A$</span>, <span class="math-container">$C&gt;D$</span>, <span class="math-container">$C&gt;A$</span> and <span class="math-container">$D&gt;A$</span>. The loss function in this case reduces to,</p>
  <p><span class="math-container">$$
  L = - \frac{1}{K\choose2} E_{(x,y_w,y_l) \in \mathcal{D}}\Big[\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))\Big]
  $$</span></p>
  <p>Hope this helps. I have created a <a href="https://cgnarendiran.github.io/blog/reinforcement-learning-from-human-feedback/" rel="nofollow noreferrer">blog post to explain RLHF in conversational AI models</a> if you want to understand better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T06:32:50.787Z
Answers.LastActivityDate: 2023-02-27T06:33:55.643Z
Answers.LastEditDate: 2023-02-27T06:33:55.643Z
Answers.UserDisplayName: cgnarendiran
Body: <p>InstructGPT: What is the sigma in the loss function and why <span class="math-container">$\log(\cdot)$</span> is being used?</p>
  <p><span class="math-container">$$ \operatorname{loss}(\theta) = -\frac{1}{\binom{K}{2}}E_{(x,y_w,y_l)\sim D}[\log(\sigma(r_{\theta}(x, y_w) - r_{\theta}(x, y_l)))] $$</span></p>
  <p>The equation was taken from the <a href="https://arxiv.org/abs/2203.02155" rel="nofollow noreferrer">InstructGPT paper</a>.</p>
  
Tags: <machine-learning><papers><open-ai><chat-bots><instruct-gpt>
Title: InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
UserDisplayName: Nathan G
ThreadText: # InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
  
  (Asked by: username_0 on 2023-01-17)
  
  InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
  
  $$ \operatorname{loss}(\theta) = -\frac{1}{\binom{K}{2}}E\_{(x,y\_w,y\_l)\sim D}[\log(\sigma(r\_{\theta}(x, y\_w) - r\_{\theta}(x, y\_l)))] $$
  
  The equation was taken from the [InstructGPT paper](https://arxiv.org/abs/2203.02155).
  
  **Tags:** machine-learning, papers, open-ai, chat-bots, instruct-gpt
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-01-27. Score: 4
  
  According to [this guide](https://wandb.ai/ayush-thakur/RLHF/reports/Understanding-Reinforcement-Learning-from-Human-Feedback-RLHF-Part-1--VmlldzoyODk5MTIx), the sigma in this formula refers to the sigmoid activation function. The guide does not tell exactly why the sigmoid function is used here, so I will try to give a full explanation of how this loss formulation works (page 8, formula 1 [in the InstructGPT paper](https://arxiv.org/abs/2203.02155)):
  
  $\text{loss}(\theta)=-\frac{1}{\binom{K}{2}}E\_{(x,y\_w,y\_l) \sim D} [log(\sigma(r\_\theta(x,y\_w)-r\_\theta(x,y\_l)))]$
  
  In the following I will use the notation from the paper:
  
  $x$ refers to the given instruction
  
  ($y\_w$, $y\_l$) refers to a pair of responses out of the list of responses which a human ranked based on their preference
  
  $y\_w$ refers to the response that is preferred over the other, lesser preferred response $y\_l$
  
  $r$ refers to the reward model
  
  $\theta$ refers to the trainable parameters of that reward model
  
  $\sigma$ refers to the sigmoid activation function.
  
  If you interpret $\sigma(r\_\theta(x, y\_w) - r\_\theta(x, y\_l))$ as *the probability that the reward model assigns a higher reward to the preferred response $y\_w$ than to the lesser preferred response $y\_l$*, the formula makes total sense. How do you maximize a probability? Exactly, by minimizing the negative log of that probability (this is called *negative log likelihood*). Hence the minus sign in front of the formula and log function around the sigmoid.
  
  In order to elaborate a bit more on that: If the reward model is working fine, then $r\_\theta(x, y\_w)$ will be a very large positive number and $r\_\theta(x, y\_l)$ will be a much lower number (maybe even a negative number). The difference $r\_\theta(x, y\_w) - r\_\theta(x, y\_l)$ will then be a very large positive number. And the sigmoid of a very large positive number approaches $1$. In that case, everything is according to plan and the loss will be very small (close to zero). However, if the reward model is failing, the assigned reward $r\_\theta(x, y\_w)$ might be much smaller than $r\_\theta(x, y\_l)$. Hence, the difference $r\_\theta(x, y\_w) - r\_\theta(x, y\_l)$ will be a (possibly very large) negative number. Take the sigmoid of that and you get a value that approaches $0$ (thus, the *probability that the reward model assigns a higher reward to the preferred response* will be small). As we are trying to maximize a probability by minimizing the negative log likelihood, we get a large loss in that case.
  
  As there will be a varying number of ranked responses for each instruction in one batch ("between $K = 4$ and
  $K = 9$ responses"), the losses of those pairwise comparisons must be weighted, so that each instruction has the same impact on the gradient update, no matter how many responses the humans have been presented for each instruction. The number of pairwise comparisons out of $K$ is $\binom{K}{2}$.
  
  In order to wrap it up: By minimizing the loss described in the paper, the reward model gets incentivized to assign a large positive reward to responses the (hopefully adequately paid *cough*) humans in front of their computers consider to be very good responses and very large negative rewards to responses which those humans consider to be very bad. And this is exactly what is desired in order to fine-tune an LLM according to human preference using reinforcement learning.
  
  > **Comments:**
  > **username_0**: Thank you. Any idea why it is actually needed here?  
  > **username_3**: To improve this answer could you summarize the specific information from the source that answers the question. This link could become invalid in the future. So by extracting the content that answers the question makes it a stronger and more durable answer for those in the future.  
  > **username_1**: NathanB and @username_3: alright, I expanded the answer in order to give a full explanation of the loss formula. This is based on the linked guide, the original InstructGPT paper, as well as my personal understanding/interpretation of the material.  
  > **username_2**: @username_1 Thank you for your response. I'm struggling to understand the quantity inside the sigmoid function: (,)−(,). If I am comparing two responses, A and B, and want to calculate the loss of B, if B is ranked higher than A, how is (,)−(,) calculated?  
  > **username_1**: @username_2 first, the loss is based on a comparison of A and B, so asking for only calculating the "loss of B" does not make much sense. You would rather say "the loss of the comparison of A and B, where B should be higher than A".  So if the human annotator said that (given the instruction $x$) the response B is preferred over A, then you assign $y_w = B$ and $y_l = A$. Then you feed $(x, y_w)$ into your reward model $r_\theta$, resulting in one scalar value (the value of the output neuron of your reward model). Repeat this for $(x, y_l)$ and you can calculate that quantity :-)  
  
  ---
  
  ### Answer by username_4 on 2023-02-27. Score: 2
  
  The probability of preferring one trajectory/response (say $A$) over the other (say $B$) is given by,
  
  $$
  \begin{aligned}
  P(A>B) &= \frac{\exp(r\_\theta(A))}{\exp(r\_\theta(A))+\exp(r\_\theta(B))}\\
  &= \frac{1}{1+\exp(-(r\_\theta(A)-r\_\theta(B)))}\\
  &= \sigma(r\_\theta(A)-r\_\theta(B))
  \end{aligned}
  $$
  
  where $\sigma(x) = \frac{1}{1+e^{-x}}$ is the sigmoid function. This follows from the [Bradley-Terry model](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model) for estimating score functions from pairwise preferences. If say $y\_w$ and $y\_l$ are the responses from the model (augmented with the query $x$), and we prefer $y\_w$ over $y\_l$, then $\mu(y\_w)=1; \mu(y\_l)=0$. We can use the following cross-entropy loss function and substitute for the probability from above equation to train our reward model:
  
  $$
  \begin{aligned}
  L &= -\mu(y\_w) \log(P(y\_w>y\_l)) + \mu(y\_l) \log(P(y\_l>y\_w))\\
  &= -\log(P(y\_w>y\_l))\\
  &= -\log(\sigma(r\_\theta(x,y\_w)-r\_\theta(x,y\_l)))
  \end{aligned}
  $$
  
  In InstructGPT, the model is made to generate $K$ responses. So we can have $K\choose2$ pairs of comparisons that we can make. Example if the model generates four responses, $A, B, C, D$ and our ranking is $B>C>D>A$, then there are ${4\choose2}=6$ comparisons possible: $B>C$, $B>D$, $B>A$, $C>D$, $C>A$ and $D>A$. The loss function in this case reduces to,
  
  $$
  L = - \frac{1}{K\choose2} E\_{(x,y\_w,y\_l) \in \mathcal{D}}\Big[\log(\sigma(r\_\theta(x,y\_w)-r\_\theta(x,y\_l)))\Big]
  $$
  
  Hope this helps. I have created a [blog post to explain RLHF in conversational AI models](https://username_4.github.io/blog/reinforcement-learning-from-human-feedback/) if you want to understand better.
  
  
  ---
  
  