Body: <p>I am following the OpenAI's spinning up tutorial <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">Part 3: Intro to Policy Optimization</a>. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the intuition behind it, I struggle to find a proof in the literature.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-10T13:55:33.057Z
Comments.Text: Does the answer to [this](https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work?rq=1) question answer yours as well?
Comments.UserDisplayName: user5093249
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-10T14:14:26.923Z
Comments.Text: No, the linked question only proofs that the reward-to-go does not introduce any bias to the gradient estimate.
Comments.UserDisplayName: sirKris van Dela
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-10T14:33:49.653Z
Comments.Text: This is nontrivial to prove, actually anything involving stochastic function approximation is nontrivial. You can search research papers, you won't find it in any book right now
Comments.UserDisplayName: FourierFlux
Tags: <reinforcement-learning><policy-gradients><proofs><reward-to-go>
Title: What is the proof that "reward-to-go" reduces variance of policy gradient?
UserDisplayName: sirKris van Dela
ThreadText: # What is the proof that "reward-to-go" reduces variance of policy gradient?
  
  (Asked by: username_0 on 2020-06-10)
  
  I am following the OpenAI's spinning up tutorial [Part 3: Intro to Policy Optimization](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html). It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the intuition behind it, I struggle to find a proof in the literature.
  
  **Tags:** reinforcement-learning, policy-gradients, proofs, reward-to-go
  > **Comments:**
  > **username_1**: Does the answer to [this](https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work?rq=1) question answer yours as well?  
  > **username_0**: No, the linked question only proofs that the reward-to-go does not introduce any bias to the gradient estimate.  
  > **username_2**: This is nontrivial to prove, actually anything involving stochastic function approximation is nontrivial. You can search research papers, you won't find it in any book right now  
  
  ---
  
  ## 0 answers
  
  