Answers.Body: <p>The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application.</p>
  <p>If you're using REINFORCE to estimate policy gradients in RL (this is the common use case), your parameters are parameterizing a policy function. The inputs are states and the outputs are actions. The issue with estimating the gradient of the parameters is as follows. In RL the objective is to maximize expected rewards. This is an expectation with respect to the policy of the cumulative reward of a trajectory. If you take the gradient of this wrt the parameters, since the expectation is wrt the policy which depends on the parameters, you can't move the gradient inside the expectation so you can't estimate the gradient from samples. Using REINFORCE however, you use the &quot;log-derivative&quot; trick to rewrite the objective as an expected gradient, which <em>can</em> be estimated from samples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-14T14:20:14.897Z
Answers.LastActivityDate: 2020-07-14T14:20:14.897Z
Answers.UserDisplayName: harwiltz
Body: <p>I've seen a few mentions in papers that neural network parameters can be found using REINFORCE algorithm. It was mentioned in the context of nondifferentiable operations involving e.g. step function which appears in <a href="https://arxiv.org/abs/1502.03044" rel="nofollow noreferrer">&quot;hard attention&quot;</a> or weight pruning. Unfortunately, I haven't seen how to really do this. In Markov Decision Process we have, states (S), actions (A) and rewards (R) so what is what in case of neural net? I don't see how can we find parameters or neural net if the gradient is not well defined.
  Any code sample or explanation?</p>
  
Tags: <reinforcement-learning><optimization><policy-gradients>
Title: How to optimize neural network parameters with REINFORCE
UserDisplayName: WoofDoggy
ThreadText: # How to optimize neural network parameters with REINFORCE
  
  (Asked by: username_0 on 2020-07-13)
  
  I've seen a few mentions in papers that neural network parameters can be found using REINFORCE algorithm. It was mentioned in the context of nondifferentiable operations involving e.g. step function which appears in ["hard attention"](https://arxiv.org/abs/1502.03044) or weight pruning. Unfortunately, I haven't seen how to really do this. In Markov Decision Process we have, states (S), actions (A) and rewards (R) so what is what in case of neural net? I don't see how can we find parameters or neural net if the gradient is not well defined.
  Any code sample or explanation?
  
  **Tags:** reinforcement-learning, optimization, policy-gradients
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-14. Score: 2
  
  The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue they're describing is not one that you should expect to find in a policy gradient application.
  
  If you're using REINFORCE to estimate policy gradients in RL (this is the common use case), your parameters are parameterizing a policy function. The inputs are states and the outputs are actions. The issue with estimating the gradient of the parameters is as follows. In RL the objective is to maximize expected rewards. This is an expectation with respect to the policy of the cumulative reward of a trajectory. If you take the gradient of this wrt the parameters, since the expectation is wrt the policy which depends on the parameters, you can't move the gradient inside the expectation so you can't estimate the gradient from samples. Using REINFORCE however, you use the "log-derivative" trick to rewrite the objective as an expected gradient, which *can* be estimated from samples.
  
  
  ---
  
  