Answers.Body: <p>REINFORCE is called a gradient estimator because it doesn't work on the true gradient, that comes from a loss function and the whole data, but makes up a heuristic loss, so that the gradient it ends up with isn't the true one. Let's see that with the REINFORCE equation:</p>
  
  <p><span class="math-container">$$
  {\huge
  \Delta \mathbf{\theta}_t = \alpha \nabla_{\mathbf{\theta}} \log \pi_{\mathbf{\theta}} (a_t \mid s_t) v_t
  }%
  $$</span></p>
  
  <p>As this shows, the gradient is still there (<span class="math-container">$\nabla_\theta$</span>). But the policy corresponds to the network's output, so we can use backpropagation to compute the gradient of that heuristic loss with respect to the weights. The real gradient is unknown to us, but this estimation will do the job.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T00:10:19.667Z
Answers.Text: I have a general understanding how policy gradient methods work, however this case is a bit different. First of all this is a supervised setting, i.e. we know the true loss at every step. What we can't compute though is the gradients of the parameters w.r.t the loss, due to the stochastic nature of the network. This is where the claim to use REINFORCE as a **gradient estimator**. I don't understand how, though...
Answers.UserDisplayName: TmBrdy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T07:43:08.510Z
Answers.Text: Ok. This shows me that I haven't explained myself well. The funny thing is that, with REINFORCE, you don't bypass the gradient computation (backpropagation), but the need of having a supervised loss. What you have in a RL scenario is a reward signal and usually very sparse. I'll try to make the answer clearer.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T23:00:25.553Z
Answers.Text: I think it would be useful to explain the terms of the REINFORCE rule and why they are needed with respect to the question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-05T00:12:52.437Z
Answers.LastActivityDate: 2019-08-06T22:59:32.010Z
Answers.LastEditDate: 2019-08-06T22:59:32.010Z
Answers.UserDisplayName: David
Body: <p>In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly).</p>
  
  <p>Some such examples are <a href="https://arxiv.org/pdf/1502.03044.pdf" rel="nofollow noreferrer">Show, Attend and Tell</a>, <a href="https://arxiv.org/abs/1406.6247" rel="nofollow noreferrer">Recurrent models of visual attention</a> and <a href="https://arxiv.org/abs/1412.7755" rel="nofollow noreferrer">Multiple Object Recognition with Visual Attention</a>.</p>
  
  <p>However, I haven't figured out how this exactly works. How do we "bypass" the gradient's computation by using the REINFORCE learning rule? Does anyone have any insight on this? </p>
  
Tags: <neural-networks><backpropagation><reinforce>
Title: How is REINFORCE used instead of Backpropagation?
UserDisplayName: TmBrdy
ThreadText: # How is REINFORCE used instead of Backpropagation?
  
  (Asked by: username_0 on 2019-08-04)
  
  In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly).
  
  Some such examples are [Show, Attend and Tell](https://arxiv.org/pdf/1502.03044.pdf), [Recurrent models of visual attention](https://arxiv.org/abs/1406.6247) and [Multiple Object Recognition with Visual Attention](https://arxiv.org/abs/1412.7755).
  
  However, I haven't figured out how this exactly works. How do we "bypass" the gradient's computation by using the REINFORCE learning rule? Does anyone have any insight on this?
  
  **Tags:** neural-networks, backpropagation, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-05. Score: 1
  
  REINFORCE is called a gradient estimator because it doesn't work on the true gradient, that comes from a loss function and the whole data, but makes up a heuristic loss, so that the gradient it ends up with isn't the true one. Let's see that with the REINFORCE equation:
  
  $$
  {\huge
  \Delta \mathbf{\theta}\_t = \alpha \nabla\_{\mathbf{\theta}} \log \pi\_{\mathbf{\theta}} (a\_t \mid s\_t) v\_t
  }%
  $$
  
  As this shows, the gradient is still there ($\nabla\_\theta$). But the policy corresponds to the network's output, so we can use backpropagation to compute the gradient of that heuristic loss with respect to the weights. The real gradient is unknown to us, but this estimation will do the job.
  
  > **Comments:**
  > **username_0**: I have a general understanding how policy gradient methods work, however this case is a bit different. First of all this is a supervised setting, i.e. we know the true loss at every step. What we can't compute though is the gradients of the parameters w.r.t the loss, due to the stochastic nature of the network. This is where the claim to use REINFORCE as a **gradient estimator**. I don't understand how, though...  
  > **username_1**: Ok. This shows me that I haven't explained myself well. The funny thing is that, with REINFORCE, you don't bypass the gradient computation (backpropagation), but the need of having a supervised loss. What you have in a RL scenario is a reward signal and usually very sparse. I'll try to make the answer clearer.  
  > **username_2**: I think it would be useful to explain the terms of the REINFORCE rule and why they are needed with respect to the question.  
  
  ---
  
  