Answers.Body: <p>For Policy-Gradient methods which require a differentiable sample from the action distribution, such as Soft Actor-Critic, then you are correct that it suffers from the same problem of requiring the gradient to be able to pass through these samples.</p>
  <p>The common way around this is to consider distributions that can be sampled from using the re-parameterisation trick. The classic example of this is with a Gaussian distribution: suppose our network outputs mean and standard deviation <span class="math-container">$\mu_\theta$</span> and <span class="math-container">$\sigma_{\theta}$</span>, respectively (<span class="math-container">$\theta$</span> are the network parameters), then we can note that <span class="math-container">$X = \mu_\theta + \sigma_\theta \times \epsilon$</span>, where <span class="math-container">$\epsilon$</span> follows a Unit-Normal distribution. So, to sample from <span class="math-container">$X$</span>, all we need to do is sample from the unit normal distribution, which does not require us to backprop through our network parameters.</p>
  <p>The same can be achieved for discrete actions using the Gumbel-Softmax trick. For information on this, I would recommend <a href="https://sassafras13.github.io/GumbelSoftmax/" rel="nofollow noreferrer">this blog-post</a> (and the references within).</p>
  <p>For algorithms derived from the REINFORCE algorithm, where the policy gradient is given by <span class="math-container">$G_t \nabla_\theta \log \pi_\theta(a|s)$</span>, then you can see that this is equivalent to maximising the (log) likelihood, scaled by the returns. All we need to do here is obtain the parameters of the distribution of our policy (e.g. mean and std-dev of a Gaussian) by doing a forward pass of the network for the current state, and then evaluating the PDF/PMF at the action we selected -- this does not require the sampled action to be differentiable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T16:28:12.463Z
Answers.Text: I think I am still very confused about something: let's say for a non-actor-critic policy gradient method, we already have a stochastic policy. I don't understand how is the policy calculated w.r.t. the outcome, since the outcome is the realization of one possibility of the action from the distribution. In other words, from a particular outcome, the outside observer wouldn't be able to know this is a stochastic policy. How to compete the gradient w.r.t. this outcome then?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T16:59:27.697Z
Answers.Text: if the answer is monte carlo, then even of the stochastic policy is a discrete categorical distribution, it doesn't matter anymore isn't it?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T21:25:21.310Z
Answers.Text: It's not totally clear to me what you mean. If not using a PG method that requires sampling from the policy, the alternatives are value learning (we don't explicitly look at the policy here, just learn a Q-function for, usually, the greedy policy), or alternatively a PG algorithm that originates from the REINFORCE update. Here, whilst we sample from the policy, we don't need to differentiate through the sampled action. We just maximise the likelihood (under the policy) of taking the action we saw, scaled by the returns.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T09:10:11.680Z
Answers.Text: 'Here, whilst we sample from the policy, we don't need to differentiate through the sampled action. We just maximise the likelihood (under the policy) of taking the action we saw, scaled by the returns.' but to update the model parameters which parameterize the policy, we still need gradient don't we?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T09:14:53.103Z
Answers.Text: Yes, but you don't need to differentiate through the sampled action. You just do a forward pass of your network to output the parameters of your distribution, and then evaluate the pdf/pmf at the action you chose and differentiate the output (remembering to also scale by the returns).
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T15:12:02.510Z
Answers.LastActivityDate: 2023-04-05T09:22:21.047Z
Answers.LastEditDate: 2023-04-05T09:22:21.047Z
Answers.UserDisplayName: David
Answers.Body: <p>You can use the reparameterization trick only when the reward is differentiable with respect to the parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-19T12:34:15.143Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-19T11:01:14.337Z
Answers.LastActivityDate: 2024-12-19T11:01:14.337Z
Answers.UserDisplayName: nn0p
Body: <p>If we want to learn a stochastic policy with the policy gradient method, we have to <em>sample</em> from the distribution to get an action.</p>
  <p>Wouldn't this lead to the same issue that variational autoencoders face without the reparameterization trick, where the gradient cannot pass through <em>sampling</em>? Or is the reparameterization trick also used in the stochastic policy gradient method to address this issue?</p>
  
Tags: <reinforcement-learning><backpropagation><policy-gradients><sampling><reparameterization-trick>
Title: Is the re-parameterization trick necessary in the policy gradient method?
UserDisplayName: Sam
ThreadText: # Is the re-parameterization trick necessary in the policy gradient method?
  
  (Asked by: username_0 on 2023-04-04)
  
  If we want to learn a stochastic policy with the policy gradient method, we have to *sample* from the distribution to get an action.
  
  Wouldn't this lead to the same issue that variational autoencoders face without the reparameterization trick, where the gradient cannot pass through *sampling*? Or is the reparameterization trick also used in the stochastic policy gradient method to address this issue?
  
  **Tags:** reinforcement-learning, backpropagation, policy-gradients, sampling, reparameterization-trick
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-04-04. Score: 2
  
  For Policy-Gradient methods which require a differentiable sample from the action distribution, such as Soft Actor-Critic, then you are correct that it suffers from the same problem of requiring the gradient to be able to pass through these samples.
  
  The common way around this is to consider distributions that can be sampled from using the re-parameterisation trick. The classic example of this is with a Gaussian distribution: suppose our network outputs mean and standard deviation $\mu\_\theta$ and $\sigma\_{\theta}$, respectively ($\theta$ are the network parameters), then we can note that $X = \mu\_\theta + \sigma\_\theta \times \epsilon$, where $\epsilon$ follows a Unit-Normal distribution. So, to sample from $X$, all we need to do is sample from the unit normal distribution, which does not require us to backprop through our network parameters.
  
  The same can be achieved for discrete actions using the Gumbel-Softmax trick. For information on this, I would recommend [this blog-post](https://sassafras13.github.io/GumbelSoftmax/) (and the references within).
  
  For algorithms derived from the REINFORCE algorithm, where the policy gradient is given by $G\_t \nabla\_\theta \log \pi\_\theta(a|s)$, then you can see that this is equivalent to maximising the (log) likelihood, scaled by the returns. All we need to do here is obtain the parameters of the distribution of our policy (e.g. mean and std-dev of a Gaussian) by doing a forward pass of the network for the current state, and then evaluating the PDF/PMF at the action we selected -- this does not require the sampled action to be differentiable.
  
  > **Comments:**
  > **username_0**: I think I am still very confused about something: let's say for a non-actor-critic policy gradient method, we already have a stochastic policy. I don't understand how is the policy calculated w.r.t. the outcome, since the outcome is the realization of one possibility of the action from the distribution. In other words, from a particular outcome, the outside observer wouldn't be able to know this is a stochastic policy. How to compete the gradient w.r.t. this outcome then?  
  > **username_0**: if the answer is monte carlo, then even of the stochastic policy is a discrete categorical distribution, it doesn't matter anymore isn't it?  
  > **username_1**: It's not totally clear to me what you mean. If not using a PG method that requires sampling from the policy, the alternatives are value learning (we don't explicitly look at the policy here, just learn a Q-function for, usually, the greedy policy), or alternatively a PG algorithm that originates from the REINFORCE update. Here, whilst we sample from the policy, we don't need to differentiate through the sampled action. We just maximise the likelihood (under the policy) of taking the action we saw, scaled by the returns.  
  > **username_0**: 'Here, whilst we sample from the policy, we don't need to differentiate through the sampled action. We just maximise the likelihood (under the policy) of taking the action we saw, scaled by the returns.' but to update the model parameters which parameterize the policy, we still need gradient don't we?  
  > **username_1**: Yes, but you don't need to differentiate through the sampled action. You just do a forward pass of your network to output the parameters of your distribution, and then evaluate the pdf/pmf at the action you chose and differentiate the output (remembering to also scale by the returns).  
  
  ---
  
  ### Answer by username_2 on 2024-12-19. Score: 0
  
  You can use the reparameterization trick only when the reward is differentiable with respect to the parameters.
  
  > **Comments:**
  > **username_3**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  