<p>This comes from SAC algorithm.</p>
<h2>Actor with REINFORCE</h2>
<p><span class="math-container">$$
\nabla_{\theta}J = \mathbb{E}_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla_{\theta}\log(\pi_{\theta} (a|s) Q_{\phi}(s,a))]
$$</span></p>
<h2>Actor with Reparameter</h2>
<p>REINFORCE works quite well with many samples, but particularly in high-dimensional action spaces, it starts
to require a lot of samples to give low variance. We can improve this by using the reparametrized gradient.Parametrize <span class="math-container">$\pi_{\theta}$</span> as <span class="math-container">$\mu_{\theta}(s) + \sigma_{\theta}(s)\epsilon$</span>, where <span class="math-container">$\epsilon$</span> is normally distributed. Then we can write:
<span class="math-container">$$
\nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi_{\theta}(a|s)}[Q(s,a)] = \nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[Q(s, \mu_{\theta}(s) + \sigma_{\theta}(s)\epsilon))] =\mathbb{E}_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[\nabla_{\theta}Q(s, \mu_{\theta}(s) + \sigma_{\theta}(s)\epsilon))]
$$</span>
This gradient estimator often gives a much lower variance, so it can be used with few samples (in practice,
just using a single sample tends to work very well).</p>
<h2>My question</h2>
<p>Why <span class="math-container">$\nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi_{\theta}(a|s)}[Q(s,a)]$</span> is a gradient estimator? To my knowledge, the <span class="math-container">$\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla_{\theta}\log(\pi_{\theta} (a|s) Q_{\phi}(s,a))]$</span> is the gradient estimator, which is not equal to the  <span class="math-container">$\nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D}, a\sim \pi_{\theta}(a|s)}[Q(s,a)]$</span> .</p>

CC BY-SA 4.0
2024-04-21T18:06:52.610Z
SAC is a different algorithm than REINFORCE. the two optimisations are not the same, hence why they are different.
David
CC BY-SA 4.0
2024-04-21T22:16:26.727Z
@David is correct, also consider that REINFOCE is on policy, SAC is not, REINFORCE works for discrete actions, SAC not... you are comparing two very different methods, you can hardly draw any trivial conclusion on why they should look the same... consider reading the Deterministic policy gradient paper from Silver, then starting from there, DDPG, TD3, and SAC, you should see the full progression from policy gradient to SAC at that point
Alberto
CC BY-SA 4.0
2024-04-22T15:07:57.283Z
@Alberto SAC does work with discrete actions.
David
CC BY-SA 4.0
2024-04-22T15:37:06.420Z
@David not the one the OP is citing (the one with the reparametrization), though you can use its estimator for a sort of REINFORCE like algorithm
Alberto
CC BY-SA 4.0
2024-04-22T17:24:02.953Z
@Alberto ah yes sorry, I forgot they specifically quoted reparam of a Normal distribution
David
CC BY-SA 4.0
2024-04-22T17:39:53.930Z
@David no prob man, you know a lot of stuff, seen you under plenty of questions with very smart observations, i can only be happy if you correct me, that can only mean that I have to learn new stuff!
Alberto
<reinforcement-learning>
Why policy gradient can be rewritten to the expection of Q values?
yeebo xie
# Why policy gradient can be rewritten to the expection of Q values?

(Asked by: username_0 on 2024-04-21)

This comes from SAC algorithm.

## Actor with REINFORCE

$$
\nabla\_{\theta}J = \mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla\_{\theta}\log(\pi\_{\theta} (a|s) Q\_{\phi}(s,a))]
$$

## Actor with Reparameter

REINFORCE works quite well with many samples, but particularly in high-dimensional action spaces, it starts
to require a lot of samples to give low variance. We can improve this by using the reparametrized gradient.Parametrize $\pi\_{\theta}$ as $\mu\_{\theta}(s) + \sigma\_{\theta}(s)\epsilon$, where $\epsilon$ is normally distributed. Then we can write:
$$
\nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi\_{\theta}(a|s)}[Q(s,a)] = \nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[Q(s, \mu\_{\theta}(s) + \sigma\_{\theta}(s)\epsilon))] =\mathbb{E}\_{s\sim \mathcal{D}, \epsilon\sim \mathcal{N}}[\nabla\_{\theta}Q(s, \mu\_{\theta}(s) + \sigma\_{\theta}(s)\epsilon))]
$$
This gradient estimator often gives a much lower variance, so it can be used with few samples (in practice,
just using a single sample tends to work very well).

## My question

Why $\nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi\_{\theta}(a|s)}[Q(s,a)]$ is a gradient estimator? To my knowledge, the $\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla\_{\theta}\log(\pi\_{\theta} (a|s) Q\_{\phi}(s,a))]$ is the gradient estimator, which is not equal to the $\nabla\_{\theta}\mathbb{E}\_{s\sim \mathcal{D}, a\sim \pi\_{\theta}(a|s)}[Q(s,a)]$ .

**Tags:** reinforcement-learning

> **Comments:**
> **username_1**: SAC is a different algorithm than REINFORCE. the two optimisations are not the same, hence why they are different.  
> **username_2**: @username_1 is correct, also consider that REINFOCE is on policy, SAC is not, REINFORCE works for discrete actions, SAC not... you are comparing two very different methods, you can hardly draw any trivial conclusion on why they should look the same... consider reading the Deterministic policy gradient paper from Silver, then starting from there, DDPG, TD3, and SAC, you should see the full progression from policy gradient to SAC at that point  
> **username_1**: @username_2 SAC does work with discrete actions.  
> **username_2**: @username_1 not the one the OP is citing (the one with the reparametrization), though you can use its estimator for a sort of REINFORCE like algorithm  
> **username_1**: @username_2 ah yes sorry, I forgot they specifically quoted reparam of a Normal distribution  
> **username_2**: @username_1 no prob man, you know a lot of stuff, seen you under plenty of questions with very smart observations, i can only be happy if you correct me, that can only mean that I have to learn new stuff!  

---

## 0 answers

