Body: <p>This comes from SAC algorithm.</p>
  <h2>Actor with REINFORCE</h2>
  <p><span class="math-container">$$
  \nabla_{\theta}J = \mathbb{E}_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla_{\theta}\log(\pi_{\theta} (a|s) Q_{\phi}(s,a))]
  $$</span></p>
  <h2>Actor with Reparameter</h2>
  <p>REINFORCE works quite well with many samples, but particularly in high-dimensional action spaces, it starts
  to require a lot of samples to give low variance. We can improve this by using the reparametrized gradient.Parametrize <span class="math-container">$\pi_{\theta}$</span> as <span class="math-container">$\mu_{\theta}(s) + \sigma_{\theta}(s)\epsilon$</span>, where <span class="math-container">$\epsilon$</span> is normally distributed. Then we can write:
  <span class="math-container">$$
  \nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi_{\theta}(a|s)}[Q(s,a)] = \nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[Q(s, \mu_{\theta}(s) + \sigma_{\theta}(s)\epsilon))] =\mathbb{E}_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[\nabla_{\theta}Q(s, \mu_{\theta}(s) + \sigma_{\theta}(s)\epsilon))]
  $$</span>
  This gradient estimator often gives a much lower variance, so it can be used with few samples (in practice,
  just using a single sample tends to work very well).</p>
  <h2>My question</h2>
  <p>Why <span class="math-container">$\nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi_{\theta}(a|s)}[Q(s,a)]$</span> is a gradient estimator? To my knowledge, the <span class="math-container">$\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla_{\theta}\log(\pi_{\theta} (a|s) Q_{\phi}(s,a))]$</span> is the gradient estimator, which is not equal to the  <span class="math-container">$\nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi_{\theta}(a|s)}[Q(s,a)]$</span> .</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-21T18:06:52.610Z
Comments.Text: SAC is a different algorithm than REINFORCE. the two optimisations are not the same, hence why they are different.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-21T22:16:26.727Z
Comments.Text: @David is correct, also consider that REINFOCE is on policy, SAC is not, REINFORCE works for discrete actions, SAC not... you are comparing two very different methods, you can hardly draw any trivial conclusion on why they should look the same... consider reading the Deterministic policy gradient paper from Silver, then starting from there, DDPG, TD3, and SAC, you should see the full progression from policy gradient to SAC at that point
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T15:07:57.283Z
Comments.Text: @Alberto SAC does work with discrete actions.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T15:37:06.420Z
Comments.Text: @David not the one the OP is citing (the one with the reparametrization), though you can use its estimator for a sort of REINFORCE like algorithm
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T17:24:02.953Z
Comments.Text: @Alberto ah yes sorry, I forgot they specifically quoted reparam of a Normal distribution
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T17:39:53.930Z
Comments.Text: @David no prob man, you know a lot of stuff, seen you under plenty of questions with very smart observations, i can only be happy if you correct me, that can only mean that I have to learn new stuff!
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning>
Title: Why policy gradient can be rewritten to the expection of Q values?
UserDisplayName: yeebo xie
ThreadText: # Why policy gradient can be rewritten to the expection of Q values?
  
  (Asked by: username_0 on 2024-04-21)
  
  This comes from SAC algorithm.
  
  ## Actor with REINFORCE
  
  $$
  \nabla\_{\theta}J = \mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla\_{\theta}\log(\pi\_{\theta} (a|s) Q\_{\phi}(s,a))]
  $$
  
  ## Actor with Reparameter
  
  REINFORCE works quite well with many samples, but particularly in high-dimensional action spaces, it starts
  to require a lot of samples to give low variance. We can improve this by using the reparametrized gradient.Parametrize $\pi\_{\theta}$ as $\mu\_{\theta}(s) + \sigma\_{\theta}(s)\epsilon$, where $\epsilon$ is normally distributed. Then we can write:
  $$
  \nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi\_{\theta}(a|s)}[Q(s,a)] = \nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[Q(s, \mu\_{\theta}(s) + \sigma\_{\theta}(s)\epsilon))] =\mathbb{E}\_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[\nabla\_{\theta}Q(s, \mu\_{\theta}(s) + \sigma\_{\theta}(s)\epsilon))]
  $$
  This gradient estimator often gives a much lower variance, so it can be used with few samples (in practice,
  just using a single sample tends to work very well).
  
  ## My question
  
  Why $\nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi\_{\theta}(a|s)}[Q(s,a)]$ is a gradient estimator? To my knowledge, the $\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla\_{\theta}\log(\pi\_{\theta} (a|s) Q\_{\phi}(s,a))]$ is the gradient estimator, which is not equal to the $\nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi\_{\theta}(a|s)}[Q(s,a)]$ .
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: SAC is a different algorithm than REINFORCE. the two optimisations are not the same, hence why they are different.  
  > **username_2**: @David is correct, also consider that REINFOCE is on policy, SAC is not, REINFORCE works for discrete actions, SAC not... you are comparing two very different methods, you can hardly draw any trivial conclusion on why they should look the same... consider reading the Deterministic policy gradient paper from Silver, then starting from there, DDPG, TD3, and SAC, you should see the full progression from policy gradient to SAC at that point  
  > **username_1**: @Alberto SAC does work with discrete actions.  
  > **username_2**: @David not the one the OP is citing (the one with the reparametrization), though you can use its estimator for a sort of REINFORCE like algorithm  
  > **username_1**: @Alberto ah yes sorry, I forgot they specifically quoted reparam of a Normal distribution  
  > **username_2**: @David no prob man, you know a lot of stuff, seen you under plenty of questions with very smart observations, i can only be happy if you correct me, that can only mean that I have to learn new stuff!  
  
  ---
  
  ## 0 answers
  
  