Answers.Body: <p>For part B, from this simple state diagram you're right we can directly compute the expected return without discounting as <span class="math-container">$\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$</span>, where <span class="math-container">$|\theta|&lt;1$</span>. Then take its gradient w.r.t <span class="math-container">$\theta$</span> we simply arrive at <span class="math-container">$1/(1-\theta)^2$</span>.</p>
  <p>For part A it seems supposed to use policy gradient theorem for a generic MDP to calculate the same, which is usually expressed as <span class="math-container">$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau \sim \pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(\tau)R(\tau)]$</span> following your reference's notation with a random trajectory <span class="math-container">$\tau$</span>. Similar to part B above to calculate the expected return, we can simplify the policy gradient as <span class="math-container">$$\nabla_{\theta}J(\theta)=\theta(1-\theta)·\nabla_{\theta}\log\theta(1-\theta)·1+\theta^2(1-\theta)·\nabla_{\theta}\log\theta^2(1-\theta)·2+...=\nabla_{\theta}\theta(1-\theta)·1+\nabla_{\theta}\theta^2(1-\theta)·2+...=\nabla_{\theta}(\theta(1-\theta)·1+\theta^2(1-\theta)·2+...)=\nabla_{\theta}(\theta/(1-\theta))$$</span>, which obviously matches the result of the above Part B. Note here my answer didn't use or need the hint of your reference since only elementary convergent geometric series sum formula and basic calculus formula involving logarithm and division are actually involved.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T10:13:14.663Z
Answers.Text: Thanks for the answer which really helps a lot!
Answers.UserDisplayName: yeebo xie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T15:13:28.773Z
Answers.Text: By MDP, do you mean Markov Decision Process? And, what does the letter J stand for?
Answers.UserDisplayName: evaristegd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T18:55:06.993Z
Answers.Text: @evaristegd J os the same as in your question, the loss
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:35:06.567Z
Answers.Text: @Alberto , thanks. Do you know where did OP got those two pictures from? Is it a book?
Answers.UserDisplayName: evaristegd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:51:39.233Z
Answers.Text: @evaristegd section 8 https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/homeworks/hw2.pdf
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T22:22:42.070Z
Answers.Text: @Alberto , thank you. How is it that you were able to find the source just by looking at the picture?
Answers.UserDisplayName: evaristegd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T08:19:47.473Z
Answers.Text: @evaristegd exact match on google, just search _"and verify that this matches the policy gradient"_ on google with quotes and you will find it
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T10:22:03.540Z
Answers.Text: May I ask, how did you come up to the computation the expected return without discounting? Why did $\theta$ get higher exponents?
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T21:55:23.433Z
Answers.Text: @Dave see the above question stated explicitly that *Do not use discounting* in bold font. Higher exponents for $\theta$ is just following multiplication rule of probability for the random trajectory $\tau$. Hope this clarifies and can get your upvote.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-27T07:19:13.533Z
Answers.Text: @cinch do you mean the joint probability formula?
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T06:58:52.850Z
Answers.LastActivityDate: 2024-03-24T07:06:16.467Z
Answers.LastEditDate: 2024-03-24T07:06:16.467Z
Answers.UserDisplayName: cinch
Body: <p>Source: CS285 2023Fall homework2 analysis.
  <a href="https://i.sstatic.net/yhKwq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yhKwq.png" alt="enter image description here" /></a></p>
  <p>In the part B, the expectation <span class="math-container">$\mathbb{E}[R(\tau)] = \theta + \theta^2 ... = \theta/(1-\theta)$</span>.
  Derivate this with respect to <span class="math-container">$\theta$</span> is <span class="math-container">$1/(1-\theta)^2$</span>.
  But how to compute the gradient using policy gradients?(Part A)</p>
  <p><a href="https://i.sstatic.net/v9E8j.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v9E8j.png" alt="enter image description here" /></a></p>
  <p>I'm really confused for computing the expectation of the gradient of log of <span class="math-container">$\pi_\theta$</span>(Trajectory).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-24T15:06:08.057Z
Comments.Text: Hi, welcome to the site! What's the source of your first picture? And the source of the second one?
Comments.UserDisplayName: evaristegd
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-25T03:08:49.713Z
Comments.Text: Both pictures come from CS285 2023Fall homework2 analysis. I added this into question. : )
Comments.UserDisplayName: yeebo xie
Tags: <reinforcement-learning>
Title: Policiy Gradient Infinite-Horizon Analysis
UserDisplayName: yeebo xie
ThreadText: # Policiy Gradient Infinite-Horizon Analysis
  
  (Asked by: username_0 on 2024-03-24)
  
  Source: CS285 2023Fall homework2 analysis.
  [![enter image description here](https://i.sstatic.net/yhKwq.png)](https://i.sstatic.net/yhKwq.png)
  
  In the part B, the expectation $\mathbb{E}[R(\tau)] = \theta + \theta^2 ... = \theta/(1-\theta)$.
  Derivate this with respect to $\theta$ is $1/(1-\theta)^2$.
  But how to compute the gradient using policy gradients?(Part A)
  
  [![enter image description here](https://i.sstatic.net/v9E8j.png)](https://i.sstatic.net/v9E8j.png)
  
  I'm really confused for computing the expectation of the gradient of log of $\pi\_\theta$(Trajectory).
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Hi, welcome to the site! What's the source of your first picture? And the source of the second one?  
  > **username_0**: Both pictures come from CS285 2023Fall homework2 analysis. I added this into question. : )  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-24. Score: 3 | Accepted answer
  
  For part B, from this simple state diagram you're right we can directly compute the expected return without discounting as $\mathbb{E}\_{\tau \sim \pi\_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$, where $|\theta|<1$. Then take its gradient w.r.t $\theta$ we simply arrive at $1/(1-\theta)^2$.
  
  For part A it seems supposed to use policy gradient theorem for a generic MDP to calculate the same, which is usually expressed as $\nabla\_{\theta}J(\theta)=\mathbb{E}\_{\tau \sim \pi\_{\theta}}[\nabla\_{\theta}\log\pi\_{\theta}(\tau)R(\tau)]$ following your reference's notation with a random trajectory $\tau$. Similar to part B above to calculate the expected return, we can simplify the policy gradient as $$\nabla\_{\theta}J(\theta)=\theta(1-\theta)·\nabla\_{\theta}\log\theta(1-\theta)·1+\theta^2(1-\theta)·\nabla\_{\theta}\log\theta^2(1-\theta)·2+...=\nabla\_{\theta}\theta(1-\theta)·1+\nabla\_{\theta}\theta^2(1-\theta)·2+...=\nabla\_{\theta}(\theta(1-\theta)·1+\theta^2(1-\theta)·2+...)=\nabla\_{\theta}(\theta/(1-\theta))$$, which obviously matches the result of the above Part B. Note here my answer didn't use or need the hint of your reference since only elementary convergent geometric series sum formula and basic calculus formula involving logarithm and division are actually involved.
  
  > **Comments:**
  > **username_0**: Thanks for the answer which really helps a lot!  
  > **username_1**: By MDP, do you mean Markov Decision Process? And, what does the letter J stand for?  
  > **username_3**: @evaristegd J os the same as in your question, the loss  
  > **username_1**: @Alberto , thanks. Do you know where did OP got those two pictures from? Is it a book?  
  > **username_3**: @evaristegd section 8 https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/homeworks/hw2.pdf  
  > **username_1**: @Alberto , thank you. How is it that you were able to find the source just by looking at the picture?  
  > **username_3**: @evaristegd exact match on google, just search _"and verify that this matches the policy gradient"_ on google with quotes and you will find it  
  > **username_4**: May I ask, how did you come up to the computation the expected return without discounting? Why did $\theta$ get higher exponents?  
  > **username_2**: @Dave see the above question stated explicitly that *Do not use discounting* in bold font. Higher exponents for $\theta$ is just following multiplication rule of probability for the random trajectory $\tau$. Hope this clarifies and can get your upvote.  
  > **username_4**: @cinch do you mean the joint probability formula?  
  
  ---
  
  