Answers.Body: <p>It's defined this way so that you can have the above equational relation between the <span class="math-container">$\eta$</span>-functions of the old and the current policies, as proved in slide 19. And on page 20, the improvement theory regrading the <span class="math-container">$\eta$</span>-function of the current policy is mentioned possibly referenced from David Silver's Reinforcement Learning lecture as the author claimed.</p>
  <p>Bear in  mind that here the <span class="math-container">$\eta$</span>-function of a policy <span class="math-container">$\pi_{\theta}$</span> is the expectation of the accumulated reward under the said (stochastic) policy, and the underlined current policy <span class="math-container">$\pi_{\theta}$</span> is required to compute the expectation of the reward term <span class="math-container">$r(s,a)$</span> under current policy in your quoted last line as clearly shown in the proof of slide 19. And <span class="math-container">$s_{t+1} \sim p$</span> is required to set the distribution of the next transitioned state <span class="math-container">$s_{t+1}$</span> following the environment's transition dynamics <span class="math-container">$p$</span>.</p>
  <p>If you have any difficulty in understanding the proof in slide 19, you're welcome to post a new question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-03T06:26:26.213Z
Answers.LastActivityDate: 2024-09-03T06:26:26.213Z
Answers.UserDisplayName: cinch
Body: <p>Please see slide 17 at <a href="https://www.syscop.de/files/2023ws/mpcrl/lecture-joschka-pg-and-ac.pdf" rel="nofollow noreferrer">https://www.syscop.de/files/2023ws/mpcrl/lecture-joschka-pg-and-ac.pdf</a></p>
  <p><a href="https://i.sstatic.net/XsJd6Ycg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XsJd6Ycg.png" alt="enter image description here" /></a></p>
  <p>Could some expert explain why the terms under expectation (underlined with red) are like that?</p>
  
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: Unclear point in definition of advantage function in PPO
UserDisplayName: DSPinfinity
ThreadText: # Unclear point in definition of advantage function in PPO
  
  (Asked by: username_0 on 2024-09-03)
  
  Please see slide 17 at <https://www.syscop.de/files/2023ws/mpcrl/lecture-joschka-pg-and-ac.pdf>
  
  [![enter image description here](https://i.sstatic.net/XsJd6Ycg.png)](https://i.sstatic.net/XsJd6Ycg.png)
  
  Could some expert explain why the terms under expectation (underlined with red) are like that?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-03. Score: 3 | Accepted answer
  
  It's defined this way so that you can have the above equational relation between the $\eta$-functions of the old and the current policies, as proved in slide 19. And on page 20, the improvement theory regrading the $\eta$-function of the current policy is mentioned possibly referenced from David Silver's Reinforcement Learning lecture as the author claimed.
  
  Bear in mind that here the $\eta$-function of a policy $\pi\_{\theta}$ is the expectation of the accumulated reward under the said (stochastic) policy, and the underlined current policy $\pi\_{\theta}$ is required to compute the expectation of the reward term $r(s,a)$ under current policy in your quoted last line as clearly shown in the proof of slide 19. And $s\_{t+1} \sim p$ is required to set the distribution of the next transitioned state $s\_{t+1}$ following the environment's transition dynamics $p$.
  
  If you have any difficulty in understanding the proof in slide 19, you're welcome to post a new question.
  
  
  ---
  
  