Body: <p>I have been recently studying Actor-Critic algorithms, and I ran into the following question.</p>
  
  <p>Let <span class="math-container">$Q_{\omega}$</span> be the critic network, and <span class="math-container">$\pi_{\theta}$</span> be the actor. It is known that in order to maximize the objective return <span class="math-container">$J(\theta)$</span>, we follow the gradient direction, which could be estimated as follows <span class="math-container">$$\nabla_{\theta}J=\mathbb{E}[Q_{\omega}(s,a).\nabla_{\theta}log \pi_{\theta} (a|s)].$$</span> But if we were to calculate the gradient of <span class="math-container">$Q^{\pi}$</span> with respect to <span class="math-container">$\theta$</span>, what are the possible approaches to do so?</p>
  
  <p><strong>More generally</strong>, say we have a network <span class="math-container">$\phi_{\omega}$</span> that is trained on data generated from another neural network, say a stochastic actor <span class="math-container">$\pi_{\theta}$</span> like in classic reinforcement learning frameworks, how to find the gradient of <span class="math-container">$\phi_{\omega}$</span> w.r.t <span class="math-container">${\theta}$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-30T09:16:25.620Z
Comments.Text: Gradient would be 0 because it doesn't depend on $\theta$.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-30T15:50:35.860Z
Comments.Text: @Brale_ I encourage you to provide a formal answer (even if short) below. If you want, you could say that the derivative is zero by showing some formulas.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-30T16:52:44.687Z
Comments.Text: @Brale_ I agree with you, but it's not that straightforward as there wasn't any clue that $\omega$ isn't a function of $\theta$, please refer to (Degris et. al, 2012) for more information.
Comments.UserDisplayName: ASA
Tags: <neural-networks><machine-learning><reinforcement-learning><policy-gradients><actor-critic-methods>
Title: What is the gradient of the Q function with respect to the policy's parameters?
UserDisplayName: ASA
ThreadText: # What is the gradient of the Q function with respect to the policy's parameters?
  
  (Asked by: username_0 on 2020-04-30)
  
  I have been recently studying Actor-Critic algorithms, and I ran into the following question.
  
  Let $Q\_{\omega}$ be the critic network, and $\pi\_{\theta}$ be the actor. It is known that in order to maximize the objective return $J(\theta)$, we follow the gradient direction, which could be estimated as follows $$\nabla\_{\theta}J=\mathbb{E}[Q\_{\omega}(s,a).\nabla\_{\theta}log \pi\_{\theta} (a|s)].$$ But if we were to calculate the gradient of $Q^{\pi}$ with respect to $\theta$, what are the possible approaches to do so?
  
  **More generally**, say we have a network $\phi\_{\omega}$ that is trained on data generated from another neural network, say a stochastic actor $\pi\_{\theta}$ like in classic reinforcement learning frameworks, how to find the gradient of $\phi\_{\omega}$ w.r.t ${\theta}$?
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning, policy-gradients, actor-critic-methods
  > **Comments:**
  > **username_1**: Gradient would be 0 because it doesn't depend on $\theta$.  
  > **username_2**: @Brale_ I encourage you to provide a formal answer (even if short) below. If you want, you could say that the derivative is zero by showing some formulas.  
  > **username_0**: @Brale_ I agree with you, but it's not that straightforward as there wasn't any clue that $\omega$ isn't a function of $\theta$, please refer to (Degris et. al, 2012) for more information.  
  
  ---
  
  ## 0 answers
  
  