Answers.Body: <p>First of all you made a mistake, equation 8 in the paper is defined with <span class="math-container">$\frac{\partial L(\theta)}{\partial s_t}$</span> not <span class="math-container">$\frac{\partial L(\theta)}{\partial\theta}$</span>.<br>
  Loss is defined as:  </p>
  
  <p><span class="math-container">$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)]$</span>  </p>
  
  <p>If we use definition of expectation (for discrete case): </p>
  
  <p><span class="math-container">$\mathbb{E}[X] = \sum\limits_{i} p_i(x_i)x_i$</span><br>
  we get  </p>
  
  <p><span class="math-container">$L(\theta) = -\sum\limits_{w^s} p_\theta(w^s)r(w^s)$</span>  </p>
  
  <p>probability <span class="math-container">$p_\theta(w_t)$</span> is defined as <span class="math-container">$softmax(s_t)$</span> = <span class="math-container">$S(s_t)$</span>. If we calculate derivative of <span class="math-container">$L(\theta)$</span> with respect to <span class="math-container">$s_t$</span>, we are basically calculating derivative of softmax <span class="math-container">$S(s_t)$</span> with respect to <span class="math-container">$s_t$</span>.  </p>
  
  <p>It can be shown and i won't derive it here that derivative of softmax function is  </p>
  
  <p><span class="math-container">$\frac{\partial S(s_i)}{\partial s_j} = S(s_i)(1_{ij} - S(s_j))$</span>  </p>
  
  <p>where <span class="math-container">$1_{ij}$</span> here is defined as yours. If we plug this derivative into derivative of  <span class="math-container">$L(\theta)$</span> and replace <span class="math-container">$S(s_t)$</span> with <span class="math-container">$p_\theta(w_t)$</span> (remember that probability distribution is defined as softmax here) we get:  </p>
  
  <p><span class="math-container">$\frac{\partial L(\theta)}{\partial s_t} = -\sum\limits_{w^s} p_\theta(w^s)(1_{w^s_t} - p_\theta(w_t))r(w^s)$</span>  </p>
  
  <p>we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:  </p>
  
  <p><span class="math-container">$\frac{\partial L(\theta)}{\partial s_t} = \mathbb{E}_{w^s \sim p_{\theta}}[(p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)]$</span>  </p>
  
  <p>now, if we ignore expectation we get:  </p>
  
  <p><span class="math-container">$\frac{\partial L(\theta)}{\partial s_t} \approx (p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)$</span>  </p>
  
  <p>I probably butchered notation is some places, but base idea should be clear</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T14:03:41.283Z
Answers.Text: Okay nice I understand, thank you! Thanks for pointing out the mistake, it was a typo & I corrected it :-)
Answers.UserDisplayName: rubencart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T14:46:38.217Z
Answers.Text: By $p_\theta(w_t)$, you mean $p_\theta(w_t^s)$?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T15:19:09.953Z
Answers.Text: I'm still confused about this:  $w^s = (w_1^s,..., w_T^s)$ is a sequence (dim $T$) of words $w^s_t$. I know & understand the $S(s)$ derivative, but here $w^s$ and $w^s_t$ do not have the same dimensions and I don't really see how you would take the softmax of $w^s$... $p_\theta(w^s_t)$ is indeed $softmax(s_t)$ I think but $p_\theta(w^s)$ should rather be a product of probabilities (one for each word in the sequence) right? Now if it were a sum or if we would drop in a logarithm the other probabilities could be omitted because we they are constant deriving w.r.t. $s_t$ maybe?
Answers.UserDisplayName: rubencart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T16:07:34.513Z
Answers.Text: I'm not entirely sure, I only glanced over the paper, I didn't actually read it. It is possible that I made some mistake especially in these middle step which they don't show in the paper. Maybe instead of $p_\theta(w^s)$  it would go $p_\theta(w_t^s)$ but they would differ in $t$, maybe you could still replace that with expectation over $w^s$ where each $w_t^s$ would be sampled
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T13:08:54.930Z
Answers.LastActivityDate: 2019-02-21T13:08:54.930Z
Answers.UserDisplayName: Brale
Body: <p>In the paper "<a href="https://arxiv.org/pdf/1612.00563.pdf" rel="nofollow noreferrer">Self-critical sequence training for image captioning</a>", on page 3, they define the loss function (of the parameters <span class="math-container">$\theta$</span>) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)): </p>
  
  <p><span class="math-container">$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)],$$</span></p>
  
  <p>where <span class="math-container">$w^s = (w_1^s,..., w_T^s)$</span> and <span class="math-container">$w^s_t$</span> is the word sampled from the model at time step <span class="math-container">$t$</span>.</p>
  
  <p>The derivation of the gradient of <span class="math-container">$L(\theta)$</span> concludes with Equation (7), where the gradient of <span class="math-container">$L(\theta)$</span> is approximated with a single sample <span class="math-container">$w^s \sim p_\theta$</span>: </p>
  
  <p><span class="math-container">$$\nabla_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown_{\theta}  log \ p_{\theta}(w^s),$$</span></p>
  
  <p>where <span class="math-container">$b$</span> is a reward baseline and <span class="math-container">$p_\theta(w^s)$</span> is the probability that sequence <span class="math-container">$w^s$</span> is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of <span class="math-container">$L(\theta)$</span> w.r.t. the input of the softmax function <span class="math-container">$s_t$</span> (final layer):</p>
  
  <p><span class="math-container">$$\nabla_{\theta}L(\theta) = \sum^T_{t=1} \frac{\partial L(\theta)}{\partial s_t} \frac{\partial s_t}{\partial \theta}$$</span> </p>
  
  <p>I still understand the equation above.</p>
  
  <p>And Equation (8):</p>
  
  <p><span class="math-container">$$\frac{\partial L(\theta)}{\partial s_t} \approx (r(w^s) - b) (p_\theta(w_t| h_t) - 1_{w^s_t}),$$</span></p>
  
  <p>where <span class="math-container">$1_{w^s_t}$</span> is <span class="math-container">$0$</span> everywhere, but <span class="math-container">$1$</span> at the <span class="math-container">$w^s_t$</span>'th entry. How do you arrive at Equation (8)? </p>
  
  <p>I'm happy to provide more information if necessary. In the paper "<a href="https://arxiv.org/pdf/1511.06732.pdf" rel="nofollow noreferrer">Sequence level training with recurrent neural networks</a>", on page 7, they derive a similar result.</p>
  
Tags: <neural-networks><reinforcement-learning><objective-functions><papers><reinforce>
Title: How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?
UserDisplayName: rubencart
ThreadText: # How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?
  
  (Asked by: username_0 on 2019-02-21)
  
  In the paper "[Self-critical sequence training for image captioning](https://arxiv.org/pdf/1612.00563.pdf)", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)):
  
  $$L(\theta) = - \mathbb{E}\_{w^s \sim p\_{\theta}}[r(w^s)],$$
  
  where $w^s = (w\_1^s,..., w\_T^s)$ and $w^s\_t$ is the word sampled from the model at time step $t$.
  
  The derivation of the gradient of $L(\theta)$ concludes with Equation (7), where the gradient of $L(\theta)$ is approximated with a single sample $w^s \sim p\_\theta$:
  
  $$\nabla\_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown\_{\theta} log \ p\_{\theta}(w^s),$$
  
  where $b$ is a reward baseline and $p\_\theta(w^s)$ is the probability that sequence $w^s$ is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of $L(\theta)$ w.r.t. the input of the softmax function $s\_t$ (final layer):
  
  $$\nabla\_{\theta}L(\theta) = \sum^T\_{t=1} \frac{\partial L(\theta)}{\partial s\_t} \frac{\partial s\_t}{\partial \theta}$$
  
  I still understand the equation above.
  
  And Equation (8):
  
  $$\frac{\partial L(\theta)}{\partial s\_t} \approx (r(w^s) - b) (p\_\theta(w\_t| h\_t) - 1\_{w^s\_t}),$$
  
  where $1\_{w^s\_t}$ is $0$ everywhere, but $1$ at the $w^s\_t$'th entry. How do you arrive at Equation (8)?
  
  I'm happy to provide more information if necessary. In the paper "[Sequence level training with recurrent neural networks](https://arxiv.org/pdf/1511.06732.pdf)", on page 7, they derive a similar result.
  
  **Tags:** neural-networks, reinforcement-learning, objective-functions, papers, reinforce
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-21. Score: 2
  
  First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s\_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$.  
  Loss is defined as:
  
  $L(\theta) = - \mathbb{E}\_{w^s \sim p\_{\theta}}[r(w^s)]$
  
  If we use definition of expectation (for discrete case):
  
  $\mathbb{E}[X] = \sum\limits\_{i} p\_i(x\_i)x\_i$  
  we get
  
  $L(\theta) = -\sum\limits\_{w^s} p\_\theta(w^s)r(w^s)$
  
  probability $p\_\theta(w\_t)$ is defined as $softmax(s\_t)$ = $S(s\_t)$. If we calculate derivative of $L(\theta)$ with respect to $s\_t$, we are basically calculating derivative of softmax $S(s\_t)$ with respect to $s\_t$.
  
  It can be shown and i won't derive it here that derivative of softmax function is
  
  $\frac{\partial S(s\_i)}{\partial s\_j} = S(s\_i)(1\_{ij} - S(s\_j))$
  
  where $1\_{ij}$ here is defined as yours. If we plug this derivative into derivative of $L(\theta)$ and replace $S(s\_t)$ with $p\_\theta(w\_t)$ (remember that probability distribution is defined as softmax here) we get:
  
  $\frac{\partial L(\theta)}{\partial s\_t} = -\sum\limits\_{w^s} p\_\theta(w^s)(1\_{w^s\_t} - p\_\theta(w\_t))r(w^s)$
  
  we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:
  
  $\frac{\partial L(\theta)}{\partial s\_t} = \mathbb{E}\_{w^s \sim p\_{\theta}}[(p\_\theta(w\_t) - 1\_{w^s\_t})(r(w^s) - b)]$
  
  now, if we ignore expectation we get:
  
  $\frac{\partial L(\theta)}{\partial s\_t} \approx (p\_\theta(w\_t) - 1\_{w^s\_t})(r(w^s) - b)$
  
  I probably butchered notation is some places, but base idea should be clear
  
  > **Comments:**
  > **username_0**: Okay nice I understand, thank you! Thanks for pointing out the mistake, it was a typo & I corrected it :-)  
  > **username_2**: By $p_\theta(w_t)$, you mean $p_\theta(w_t^s)$?  
  > **username_0**: I'm still confused about this:  $w^s = (w_1^s,..., w_T^s)$ is a sequence (dim $T$) of words $w^s_t$. I know & understand the $S(s)$ derivative, but here $w^s$ and $w^s_t$ do not have the same dimensions and I don't really see how you would take the softmax of $w^s$... $p_\theta(w^s_t)$ is indeed $softmax(s_t)$ I think but $p_\theta(w^s)$ should rather be a product of probabilities (one for each word in the sequence) right? Now if it were a sum or if we would drop in a logarithm the other probabilities could be omitted because we they are constant deriving w.r.t. $s_t$ maybe?  
  > **username_1**: I'm not entirely sure, I only glanced over the paper, I didn't actually read it. It is possible that I made some mistake especially in these middle step which they don't show in the paper. Maybe instead of $p_\theta(w^s)$  it would go $p_\theta(w_t^s)$ but they would differ in $t$, maybe you could still replace that with expectation over $w^s$ where each $w_t^s$ would be sampled  
  
  ---
  
  