Body: <p>Currently I am reading the OpenAI spinning up document about policy gradient and actor-critic method. In this webpage,<a href="https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof2.html#proof-for-using-q-function-in-policy-gradient-formula" rel="nofollow noreferrer">replace Return with Action value</a>, I think they are trying to prove that the following two formulas are mathematically equivalent.<a href="https://i.sstatic.net/y1juOr0w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/y1juOr0w.png" alt="REINFOECE VS ACTOR CRITIC" /></a>,</p>
  <p>I could not understand the derivation in that webpage. Is there any other resources for a clearer derivation for replacing the return <span class="math-container">$R(\tau)$</span> with the corresponding action value <span class="math-container">$Q^{\pi_{\theta}}(s_t,a_t)$</span>? Thank you very much!</p>
  <p><a href="https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f" rel="nofollow noreferrer">This article presents a different derivation.</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-14T19:12:41.117Z
Comments.Text: the expectation of Q wrt the policy is V, and V is the expected return R
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-14T19:54:30.423Z
Comments.Text: Read the Sutton and Barto derivation
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-15T20:19:14.483Z
Comments.Text: I find another explanation which i think is very good.  [link](https://medium.com/@guoshuai1998/i-think-your-decomposing-of-the-expectation-is-wrong-e30da5efd8f)
Comments.UserDisplayName: jim1124
Tags: <reinforcement-learning><q-learning><policy-gradients><actor-critic-methods>
Title: Proof for Using Q-Function in Policy Gradient Formula
UserDisplayName: jim1124
ThreadText: # Proof for Using Q-Function in Policy Gradient Formula
  
  (Asked by: username_0 on 2024-08-14)
  
  Currently I am reading the OpenAI spinning up document about policy gradient and actor-critic method. In this webpage,[replace Return with Action value](https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof2.html#proof-for-using-q-function-in-policy-gradient-formula), I think they are trying to prove that the following two formulas are mathematically equivalent.[![REINFOECE VS ACTOR CRITIC](https://i.sstatic.net/y1juOr0w.png)](https://i.sstatic.net/y1juOr0w.png),
  
  I could not understand the derivation in that webpage. Is there any other resources for a clearer derivation for replacing the return $R(\tau)$ with the corresponding action value $Q^{\pi\_{\theta}}(s\_t,a\_t)$? Thank you very much!
  
  [This article presents a different derivation.](https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f)
  
  **Tags:** reinforcement-learning, q-learning, policy-gradients, actor-critic-methods
  > **Comments:**
  > **username_1**: the expectation of Q wrt the policy is V, and V is the expected return R  
  > **username_2**: Read the Sutton and Barto derivation  
  > **username_0**: I find another explanation which i think is very good.  [link](https://medium.com/@guoshuai1998/i-think-your-decomposing-of-the-expectation-is-wrong-e30da5efd8f)  
  
  ---
  
  ## 0 answers
  
  