Answers.Body: <p>This is more meant like a comment to the previous answer. I also originally thought that</p>
  <p><span class="math-container">$$ \nabla_{\theta}\log \pi_{\theta}(f_{\theta}(\varepsilon, s)\mid s) = \nabla_{a}\log\pi_{\theta}(a\mid s)\vert_{a=f_{\theta}(\varepsilon,s)}\nabla_{\theta}f_{\theta}(\varepsilon, s), $$</span> instead of</p>
  <p><span class="math-container">$$ \nabla_{\theta}\log \pi_{\theta}(f_{\theta}(\varepsilon, s)\mid s) = \nabla_{a}\log\pi_{\theta}(a\mid s)\vert_{a=f_{\theta}(\varepsilon,s)}\nabla_{\theta}f_{\theta}(\varepsilon, s) + \nabla_{\theta}\log\pi_{\theta}(a\mid s)\vert_{a=f_{\theta}(\varepsilon, s)}. $$</span>
  The following is certainly less elegant, but I hope that it gives some additional intuition why we need to take the gradient with respect to <span class="math-container">$a$</span> and <span class="math-container">$\theta$</span>. For simplicity, I will assume that <span class="math-container">$a$</span> is one-dimensional, but the same argument would apply for higher dimensions. In the SAC paper, they assume that <span class="math-container">$\pi_{\theta}$</span> is a Gaussian distribution <span class="math-container">$\mathcal{N}(\mu_{\theta}(s), \sigma_{\theta}(s))$</span>. Therefore:
  <span class="math-container">$$ \log\pi_{\theta}(a\mid s)=-\frac{1}{2}\log(2\pi) - \log\sigma_{\theta}(s)-\frac{(a-\mu_{\theta}(s))^2}{2\sigma_{\theta}(s)^2}. $$</span>
  Then the gradient becomes:
  <span class="math-container">\begin{align} \nabla_{\theta}\log \pi_{\theta}(f_{\theta}(\varepsilon, s)\mid s)&amp;=-\frac{\nabla_{\theta}\sigma_{\theta}(s)}{\sigma_{\theta}(s)}-\frac{(f_{\theta}(\varepsilon, s)-\mu_{\theta}(s))(\nabla_{\theta}f_{\theta}(\varepsilon,s)-\nabla_{\theta}\mu_{\theta}(s))}{\sigma_{\theta}(s)^2} \\&amp;+\frac{(f_{\theta}(\varepsilon, s)-\mu_{\theta}(s))^2\nabla_{\theta}\sigma_{\theta}(s)}{\sigma_{\theta}(s)^3}.\end{align}</span></p>
  <p>Let us calculate now the terms on the right hand side:</p>
  <p><span class="math-container">$$ \nabla_{a}\log\pi_{\theta}(a\mid s)\vert_{a=f_{\theta}(\varepsilon,s)}=-\frac{f_{\theta}(\varepsilon,s)-\mu_{\theta}(s)}{\sigma_{\theta}(s)^2} $$</span></p>
  <p>and <span class="math-container">\begin{align}\nabla_{\theta}\log\pi_{\theta}(a\mid s)\vert_{a=f_{\theta}(\varepsilon, s)}&amp;=-\frac{\nabla_{\theta}\sigma_{\theta}(s)}{\sigma_{\theta}(s)}+\frac{(f_{\theta}(\varepsilon, s)-\mu_{\theta}(s))\nabla_{\theta}\mu_{\theta}(s)}{\sigma_{\theta}(s)^2}\\ &amp;+\frac{(f_{\theta}(\varepsilon, s)-\mu_{\theta}(s))^2\nabla_{\theta}\sigma_{\theta}(s)}{\sigma_{\theta}(s)^3},\end{align}</span>
  which proves the equality. For the Q-function, we apply the chain rule as usual as <span class="math-container">$Q$</span> does not depend on <span class="math-container">$\theta$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-30T20:28:50.303Z
Answers.LastActivityDate: 2020-09-30T20:28:50.303Z
Answers.UserDisplayName: matorbi
Answers.Body: <p>I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me.<br>
  I'll disregard expectation here to make things simpler. First, note that policy <span class="math-container">$\pi$</span> depends on parameter vector <span class="math-container">$\phi$</span> and function <span class="math-container">$f_\phi(\epsilon_t;s_t)$</span>, and value function <span class="math-container">$Q$</span> depends on parameter vector <span class="math-container">$\theta$</span> and same function <span class="math-container">$f_\phi(\epsilon_t;s_t)$</span>. Also, one important thing that authors mention in the paper and you didn't mention is that this solution is approximate gradient not the true gradient.<br>
  Our goal is to calculate gradient of objective function <span class="math-container">$J_\pi$</span> with respect to <span class="math-container">$\phi$</span>, so disregarding the expectation we have:  </p>
  
  <p><span class="math-container">$\nabla_\phi J_\pi (\phi) = \nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) - \nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t))$</span>  </p>
  
  <p>Let's see the gradient of first term on right hand side. To get the full gradient we need to calculate derivative w.r.t to both variables, <span class="math-container">$\phi$</span> and <span class="math-container">$f_\phi (\epsilon_t;s_t)$</span>, so we have:  </p>
  
  <p><span class="math-container">$\nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) = \frac {\partial \log\pi(\phi,f_\phi (\epsilon_t;s_t))}{\partial \phi} + \frac{\partial \log\pi(\phi,f_\phi (\epsilon_t;s_t))}{\partial f_\phi(\epsilon_t;s_t)} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$</span></p>
  
  <p>This is where approximation comes, they replace <span class="math-container">$f_\phi (\epsilon_t;s_t)$</span> with <span class="math-container">$a_t$</span> in some places and we have:  </p>
  
  <p><span class="math-container">$\nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) \approx \frac {\partial \log\pi(\phi,a_t)}{\partial \phi} + \frac{\partial \log\pi(\phi,a_t)}{\partial a_t} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$</span><br>
  <span class="math-container">$\nabla_\phi \log\pi(\phi,f_\phi (\epsilon_t;s_t)) \approx \nabla_\phi \log\pi(\phi,a_t) + \nabla_{a_t} \log\pi(\phi,a_t) \nabla_\phi f_\phi (\epsilon_t;s_t)$</span>  </p>
  
  <p>For the second term in first expression on right hand side we have:  </p>
  
  <p><span class="math-container">$\nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t)) = \frac {\partial Q(s_t,\theta,f_\phi (\epsilon_t;s_t))}{\partial \phi} + \frac{\partial Q(s_t,\theta,f_\phi (\epsilon_t;s_t))}{\partial f_\phi(\epsilon_t;s_t)} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$</span><br>
  <span class="math-container">$\nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t)) \approx \frac {\partial Q(s_t,\theta,a_t)}{\partial \phi} + \frac{\partial Q(s_t,\theta,a_t)}{\partial a_t} \frac{\partial f_\phi(\epsilon_t;s_t)}{\partial \phi}$</span>  </p>
  
  <p>Fist term on right hand side is 0 because <span class="math-container">$Q$</span> does not depend on <span class="math-container">$\phi$</span> so we have:  </p>
  
  <p><span class="math-container">$\nabla_\phi Q(s_t,\theta,f_\phi (\epsilon_t;s_t)) \approx \nabla_{a_t}Q(s_t, \theta,a_t)\nabla_\phi f_\phi(\epsilon_t;s_t)$</span>  </p>
  
  <p>Now you add up things and you get the final result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T15:31:47.633Z
Answers.Text: What does the notation $\pi_\phi(\cdot \mid s_t)$ actually mean in the original question? Why the $\cdot$? Why not $\pi_\phi(a_t \mid s_t)$?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T16:24:08.253Z
Answers.Text: I think they put the dot because when they wrote expectation they did $\mathbb E_{s_t\sim \mathcal D}$, they didn't put in the subscript from what distribution is the action sampled so they put the dot as a placeholder for action.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T02:13:42.743Z
Answers.Text: Thanks, @Brale_. This makes sense. I did not realize that $\pi$ would depend on $\phi$ beyond $f_\phi$.
Answers.UserDisplayName: Maybe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T15:09:59.653Z
Answers.LastActivityDate: 2019-02-14T15:30:03.607Z
Answers.LastEditDate: 2019-02-14T15:30:03.607Z
Answers.UserDisplayName: Brale
Body: <p>In the paper <a href="https://arxiv.org/pdf/1801.01290.pdf" rel="nofollow noreferrer">Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor</a>, they define the loss function for the policy network as</p>
  <p><span class="math-container">$$
  J_\pi(\phi)=\mathbb E_{s_t\sim \mathcal D}\left[D_{KL}\left(\pi_\phi(\cdot|s_t)\Big\Vert {\exp(Q_\theta(s_t,\cdot)\over Z_\theta(s_t)}\right)\right]
  $$</span></p>
  <p>Applying the reparameterization trick, let <span class="math-container">$a_t=f_\phi(\epsilon_t;s_t)$</span>, then the objective could be rewritten as</p>
  <p><span class="math-container">$$
  J_\pi(\phi)=\mathbb E_{s_t\sim \mathcal D, \epsilon \sim\mathcal N}[\log \pi_\phi(f_\phi(\epsilon_;s_t)|s_t)-Q_\theta(s_t,f_\phi(\epsilon_t;s_t))]
  $$</span></p>
  <p>They compute the gradient of the above objective as follows</p>
  <p><span class="math-container">$$
  \nabla_\phi J_\pi(\phi)=\nabla_\phi\log\pi_\phi(a_t|s_t)+(\nabla_{a_t}\log\pi_\phi(a_t|s_t)-\nabla_{a_t}Q(s_t,a_t))\nabla_\phi f_\phi(\epsilon_t;s_t)
  $$</span></p>
  <p>The thing confuses me is the first term in the gradient, where does it come from? To my best knowledge, the second large term is already the gradient we need, why do they add the first term?</p>
  
Tags: <reinforcement-learning><papers><gradient-descent><actor-critic-methods><soft-actor-critic>
Title: What is the gradient of the objective function in the Soft Actor-Critic paper?
UserDisplayName: Maybe
ThreadText: # What is the gradient of the objective function in the Soft Actor-Critic paper?
  
  (Asked by: username_0 on 2019-02-13)
  
  In the paper [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://arxiv.org/pdf/1801.01290.pdf), they define the loss function for the policy network as
  
  $$
  J\_\pi(\phi)=\mathbb E\_{s\_t\sim \mathcal D}\left[D\_{KL}\left(\pi\_\phi(\cdot|s\_t)\Big\Vert {\exp(Q\_\theta(s\_t,\cdot)\over Z\_\theta(s\_t)}\right)\right]
  $$
  
  Applying the reparameterization trick, let $a\_t=f\_\phi(\epsilon\_t;s\_t)$, then the objective could be rewritten as
  
  $$
  J\_\pi(\phi)=\mathbb E\_{s\_t\sim \mathcal D, \epsilon \sim\mathcal N}[\log \pi\_\phi(f\_\phi(\epsilon\_;s\_t)|s\_t)-Q\_\theta(s\_t,f\_\phi(\epsilon\_t;s\_t))]
  $$
  
  They compute the gradient of the above objective as follows
  
  $$
  \nabla\_\phi J\_\pi(\phi)=\nabla\_\phi\log\pi\_\phi(a\_t|s\_t)+(\nabla\_{a\_t}\log\pi\_\phi(a\_t|s\_t)-\nabla\_{a\_t}Q(s\_t,a\_t))\nabla\_\phi f\_\phi(\epsilon\_t;s\_t)
  $$
  
  The thing confuses me is the first term in the gradient, where does it come from? To my best knowledge, the second large term is already the gradient we need, why do they add the first term?
  
  **Tags:** reinforcement-learning, papers, gradient-descent, actor-critic-methods, soft-actor-critic
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-02-14. Score: 4 | Accepted answer
  
  I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me.  
  I'll disregard expectation here to make things simpler. First, note that policy $\pi$ depends on parameter vector $\phi$ and function $f\_\phi(\epsilon\_t;s\_t)$, and value function $Q$ depends on parameter vector $\theta$ and same function $f\_\phi(\epsilon\_t;s\_t)$. Also, one important thing that authors mention in the paper and you didn't mention is that this solution is approximate gradient not the true gradient.  
  Our goal is to calculate gradient of objective function $J\_\pi$ with respect to $\phi$, so disregarding the expectation we have:
  
  $\nabla\_\phi J\_\pi (\phi) = \nabla\_\phi \log\pi(\phi,f\_\phi (\epsilon\_t;s\_t)) - \nabla\_\phi Q(s\_t,\theta,f\_\phi (\epsilon\_t;s\_t))$
  
  Let's see the gradient of first term on right hand side. To get the full gradient we need to calculate derivative w.r.t to both variables, $\phi$ and $f\_\phi (\epsilon\_t;s\_t)$, so we have:
  
  $\nabla\_\phi \log\pi(\phi,f\_\phi (\epsilon\_t;s\_t)) = \frac {\partial \log\pi(\phi,f\_\phi (\epsilon\_t;s\_t))}{\partial \phi} + \frac{\partial \log\pi(\phi,f\_\phi (\epsilon\_t;s\_t))}{\partial f\_\phi(\epsilon\_t;s\_t)} \frac{\partial f\_\phi(\epsilon\_t;s\_t)}{\partial \phi}$
  
  This is where approximation comes, they replace $f\_\phi (\epsilon\_t;s\_t)$ with $a\_t$ in some places and we have:
  
  $\nabla\_\phi \log\pi(\phi,f\_\phi (\epsilon\_t;s\_t)) \approx \frac {\partial \log\pi(\phi,a\_t)}{\partial \phi} + \frac{\partial \log\pi(\phi,a\_t)}{\partial a\_t} \frac{\partial f\_\phi(\epsilon\_t;s\_t)}{\partial \phi}$  
  $\nabla\_\phi \log\pi(\phi,f\_\phi (\epsilon\_t;s\_t)) \approx \nabla\_\phi \log\pi(\phi,a\_t) + \nabla\_{a\_t} \log\pi(\phi,a\_t) \nabla\_\phi f\_\phi (\epsilon\_t;s\_t)$
  
  For the second term in first expression on right hand side we have:
  
  $\nabla\_\phi Q(s\_t,\theta,f\_\phi (\epsilon\_t;s\_t)) = \frac {\partial Q(s\_t,\theta,f\_\phi (\epsilon\_t;s\_t))}{\partial \phi} + \frac{\partial Q(s\_t,\theta,f\_\phi (\epsilon\_t;s\_t))}{\partial f\_\phi(\epsilon\_t;s\_t)} \frac{\partial f\_\phi(\epsilon\_t;s\_t)}{\partial \phi}$  
  $\nabla\_\phi Q(s\_t,\theta,f\_\phi (\epsilon\_t;s\_t)) \approx \frac {\partial Q(s\_t,\theta,a\_t)}{\partial \phi} + \frac{\partial Q(s\_t,\theta,a\_t)}{\partial a\_t} \frac{\partial f\_\phi(\epsilon\_t;s\_t)}{\partial \phi}$
  
  Fist term on right hand side is 0 because $Q$ does not depend on $\phi$ so we have:
  
  $\nabla\_\phi Q(s\_t,\theta,f\_\phi (\epsilon\_t;s\_t)) \approx \nabla\_{a\_t}Q(s\_t, \theta,a\_t)\nabla\_\phi f\_\phi(\epsilon\_t;s\_t)$
  
  Now you add up things and you get the final result.
  
  > **Comments:**
  > **username_3**: What does the notation $\pi_\phi(\cdot \mid s_t)$ actually mean in the original question? Why the $\cdot$? Why not $\pi_\phi(a_t \mid s_t)$?  
  > **username_2**: I think they put the dot because when they wrote expectation they did $\mathbb E_{s_t\sim \mathcal D}$, they didn't put in the subscript from what distribution is the action sampled so they put the dot as a placeholder for action.  
  > **username_0**: Thanks, @Brale_. This makes sense. I did not realize that $\pi$ would depend on $\phi$ beyond $f_\phi$.  
  
  ---
  
  ### Answer by username_1 on 2020-09-30. Score: 1
  
  This is more meant like a comment to the previous answer. I also originally thought that
  
  $$ \nabla\_{\theta}\log \pi\_{\theta}(f\_{\theta}(\varepsilon, s)\mid s) = \nabla\_{a}\log\pi\_{\theta}(a\mid s)\vert\_{a=f\_{\theta}(\varepsilon,s)}\nabla\_{\theta}f\_{\theta}(\varepsilon, s), $$ instead of
  
  $$ \nabla\_{\theta}\log \pi\_{\theta}(f\_{\theta}(\varepsilon, s)\mid s) = \nabla\_{a}\log\pi\_{\theta}(a\mid s)\vert\_{a=f\_{\theta}(\varepsilon,s)}\nabla\_{\theta}f\_{\theta}(\varepsilon, s) + \nabla\_{\theta}\log\pi\_{\theta}(a\mid s)\vert\_{a=f\_{\theta}(\varepsilon, s)}. $$
  The following is certainly less elegant, but I hope that it gives some additional intuition why we need to take the gradient with respect to $a$ and $\theta$. For simplicity, I will assume that $a$ is one-dimensional, but the same argument would apply for higher dimensions. In the SAC paper, they assume that $\pi\_{\theta}$ is a Gaussian distribution $\mathcal{N}(\mu\_{\theta}(s), \sigma\_{\theta}(s))$. Therefore:
  $$ \log\pi\_{\theta}(a\mid s)=-\frac{1}{2}\log(2\pi) - \log\sigma\_{\theta}(s)-\frac{(a-\mu\_{\theta}(s))^2}{2\sigma\_{\theta}(s)^2}. $$
  Then the gradient becomes:
  \begin{align} \nabla\_{\theta}\log \pi\_{\theta}(f\_{\theta}(\varepsilon, s)\mid s)&=-\frac{\nabla\_{\theta}\sigma\_{\theta}(s)}{\sigma\_{\theta}(s)}-\frac{(f\_{\theta}(\varepsilon, s)-\mu\_{\theta}(s))(\nabla\_{\theta}f\_{\theta}(\varepsilon,s)-\nabla\_{\theta}\mu\_{\theta}(s))}{\sigma\_{\theta}(s)^2} \\&+\frac{(f\_{\theta}(\varepsilon, s)-\mu\_{\theta}(s))^2\nabla\_{\theta}\sigma\_{\theta}(s)}{\sigma\_{\theta}(s)^3}.\end{align}
  
  Let us calculate now the terms on the right hand side:
  
  $$ \nabla\_{a}\log\pi\_{\theta}(a\mid s)\vert\_{a=f\_{\theta}(\varepsilon,s)}=-\frac{f\_{\theta}(\varepsilon,s)-\mu\_{\theta}(s)}{\sigma\_{\theta}(s)^2} $$
  
  and \begin{align}\nabla\_{\theta}\log\pi\_{\theta}(a\mid s)\vert\_{a=f\_{\theta}(\varepsilon, s)}&=-\frac{\nabla\_{\theta}\sigma\_{\theta}(s)}{\sigma\_{\theta}(s)}+\frac{(f\_{\theta}(\varepsilon, s)-\mu\_{\theta}(s))\nabla\_{\theta}\mu\_{\theta}(s)}{\sigma\_{\theta}(s)^2}\\ &+\frac{(f\_{\theta}(\varepsilon, s)-\mu\_{\theta}(s))^2\nabla\_{\theta}\sigma\_{\theta}(s)}{\sigma\_{\theta}(s)^3},\end{align}
  which proves the equality. For the Q-function, we apply the chain rule as usual as $Q$ does not depend on $\theta$.
  
  
  ---
  
  