Body: <p>This comes from CS285 2023Fall homework2.
  I'm self-learning RL, but analysis question seems a little hard for me.</p>
  <p><a href="https://i.sstatic.net/GLdXa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GLdXa.png" alt="enter image description here" /></a>
  In question one, we could get
  <span class="math-container">$\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$</span>, where <span class="math-container">$|\theta|&lt;1$</span>. Then take its gradient w.r.t <span class="math-container">$\theta$</span> we simply arrive at <span class="math-container">$1/(1-\theta)^2$</span>.
  \</p>
  <p>But Question 2 is still confusing me.
  The variance I compute as fllowed:
  <span class="math-container">$$Variance = \sqrt{\sum_{t=1}^{\infty } ((t-1)\theta^{t-1}(1-\theta)\nabla_{\theta }\ln \theta^{t-1}(1-\theta ) - \frac{1}{(1-\theta)^2})^2}$$</span>
  But how to simplify it to eyeball the min/max?</p>
  <p>Q3:I compute policy gradient use reward-to-go as followed, but I don't know why this is unbiased to <span class="math-container">$1/(1-\theta)^2$</span>. And How to compute the variance...
  <span class="math-container">$$
  \nabla_{\theta}J_{\theta}=\sum_{T=1}^{\infty}\theta^{T-1}(1-\theta)\sum_{t=1}^{T-1}\nabla_{\theta}\ln\theta(T-t)
  $$</span>
  <span class="math-container">$$
  = \sum_{T=1}^{\infty}\theta^{T-1}(1-\theta)\frac{T^2 - T}{2\theta}
  $$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-26T21:00:32.283Z
Comments.Text: Does this answer your question? [Policiy Gradient Infinite-Horizon Analysis](https://ai.stackexchange.com/questions/45218/policiy-gradient-infinite-horizon-analysis)
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-27T02:12:31.157Z
Comments.Text: The question was also queried by me. That question below only focus on the Q1 part and this question focus on Q2 and Q2.
Comments.UserDisplayName: yeebo xie
Tags: <reinforcement-learning>
Title: Computing variance of gradient of infinite horizon MDP
UserDisplayName: yeebo xie
ThreadText: # Computing variance of gradient of infinite horizon MDP
  
  (Asked by: username_0 on 2024-03-25)
  
  This comes from CS285 2023Fall homework2.
  I'm self-learning RL, but analysis question seems a little hard for me.
  
  [![enter image description here](https://i.sstatic.net/GLdXa.png)](https://i.sstatic.net/GLdXa.png)
  In question one, we could get
  $\mathbb{E}\_{\tau \sim \pi\_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$, where $|\theta|<1$. Then take its gradient w.r.t $\theta$ we simply arrive at $1/(1-\theta)^2$.
  \
  
  But Question 2 is still confusing me.
  The variance I compute as fllowed:
  $$Variance = \sqrt{\sum\_{t=1}^{\infty } ((t-1)\theta^{t-1}(1-\theta)\nabla\_{\theta }\ln \theta^{t-1}(1-\theta ) - \frac{1}{(1-\theta)^2})^2}$$
  But how to simplify it to eyeball the min/max?
  
  Q3:I compute policy gradient use reward-to-go as followed, but I don't know why this is unbiased to $1/(1-\theta)^2$. And How to compute the variance...
  $$
  \nabla\_{\theta}J\_{\theta}=\sum\_{T=1}^{\infty}\theta^{T-1}(1-\theta)\sum\_{t=1}^{T-1}\nabla\_{\theta}\ln\theta(T-t)
  $$
  $$
  = \sum\_{T=1}^{\infty}\theta^{T-1}(1-\theta)\frac{T^2 - T}{2\theta}
  $$
  
  **Tags:** reinforcement-learning
  
  > **Comments:**
  > **username_1**: Does this answer your question? [Policiy Gradient Infinite-Horizon Analysis](https://ai.stackexchange.com/questions/45218/policiy-gradient-infinite-horizon-analysis)  
  > **username_0**: The question was also queried by me. That question below only focus on the Q1 part and this question focus on Q2 and Q2.  
  
  ---
  
  ## 0 answers
  
  