<p>This comes from CS285 2023Fall homework2.
I'm self-learning RL, but analysis question seems a little hard for me.</p>
<p><a href="https://i.sstatic.net/GLdXa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GLdXa.png" alt="enter image description here" /></a>
In question one, we could get
<span class="math-container">$\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$</span>, where <span class="math-container">$|\theta|&lt;1$</span>. Then take its gradient w.r.t <span class="math-container">$\theta$</span> we simply arrive at <span class="math-container">$1/(1-\theta)^2$</span>.
\</p>
<p>But Question 2 is still confusing me.
The variance I compute as fllowed:
<span class="math-container">$$Variance = \sqrt{\sum_{t=1}^{\infty } ((t-1)\theta^{t-1}(1-\theta)\nabla_{\theta }\ln \theta^{t-1}(1-\theta ) - \frac{1}{(1-\theta)^2})^2}$$</span>
But how to simplify it to eyeball the min/max?</p>
<p>Q3:I compute policy gradient use reward-to-go as followed, but I don't know why this is unbiased to <span class="math-container">$1/(1-\theta)^2$</span>. And How to compute the variance...
<span class="math-container">$$
\nabla_{\theta}J_{\theta}=\sum_{T=1}^{\infty}\theta^{T-1}(1-\theta)\sum_{t=1}^{T-1}\nabla_{\theta}\ln\theta(T-t)
$$</span>
<span class="math-container">$$
= \sum_{T=1}^{\infty}\theta^{T-1}(1-\theta)\frac{T^2 - T}{2\theta}
$$</span></p>

CC BY-SA 4.0
2024-03-26T21:00:32.283Z
Does this answer your question? [Policiy Gradient Infinite-Horizon Analysis](https://ai.stackexchange.com/questions/45218/policiy-gradient-infinite-horizon-analysis)
Dave
CC BY-SA 4.0
2024-03-27T02:12:31.157Z
The question was also queried by me. That question below only focus on the Q1 part and this question focus on Q2 and Q2.
yeebo xie
<reinforcement-learning>
Computing variance of gradient of infinite horizon MDP
yeebo xie
# Computing variance of gradient of infinite horizon MDP

(Asked by: username_0 on 2024-03-25)

This comes from CS285 2023Fall homework2.
I'm self-learning RL, but analysis question seems a little hard for me.

[![enter image description here](https://i.sstatic.net/GLdXa.png)](https://i.sstatic.net/GLdXa.png)
In question one, we could get
$\mathbb{E}\_{\tau \sim \pi\_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$, where $|\theta|<1$. Then take its gradient w.r.t $\theta$ we simply arrive at $1/(1-\theta)^2$.
\

But Question 2 is still confusing me.
The variance I compute as fllowed:
$$Variance = \sqrt{\sum\_{t=1}^{\infty } ((t-1)\theta^{t-1}(1-\theta)\nabla\_{\theta }\ln \theta^{t-1}(1-\theta ) - \frac{1}{(1-\theta)^2})^2}$$
But how to simplify it to eyeball the min/max?

Q3:I compute policy gradient use reward-to-go as followed, but I don't know why this is unbiased to $1/(1-\theta)^2$. And How to compute the variance...
$$
\nabla\_{\theta}J\_{\theta}=\sum\_{T=1}^{\infty}\theta^{T-1}(1-\theta)\sum\_{t=1}^{T-1}\nabla\_{\theta}\ln\theta(T-t)
$$
$$
= \sum\_{T=1}^{\infty}\theta^{T-1}(1-\theta)\frac{T^2 - T}{2\theta}
$$

**Tags:** reinforcement-learning

> **Comments:**
> **username_1**: Does this answer your question? [Policiy Gradient Infinite-Horizon Analysis](https://ai.stackexchange.com/questions/45218/policiy-gradient-infinite-horizon-analysis)  
> **username_0**: The question was also queried by me. That question below only focus on the Q1 part and this question focus on Q2 and Q2.  

---

## 0 answers

