Answers.Body: <p>The first line of equation (11.4) is copied from equation (11.2) where the definition of your book's advantage function <span class="math-container">$\hat A_{\pi}(s_t,a_t)$</span> plays a key role and it seems you have no problem for this.</p>
  <p>The second line could be understood first rewriting the expectation wrt the random trajectory <span class="math-container">$\tau$</span> under the new policy <span class="math-container">$\pi'$</span> as the joint expectation of the random state <span class="math-container">$s$</span> and the random action <span class="math-container">$a$</span> following the distribution of <span class="math-container">$d_{\pi'}$</span> and <span class="math-container">$\pi'$</span>, respectively, since any reward trajectory is jointly decided by state and action with the environment's transition dynamics which doesn't effect advantage function at all. The author also explicitly gives the definition of <span class="math-container">$d_{\pi}$</span> (thus <span class="math-container">$d_{\pi'}$</span> implicitly) at the end of your quote which is the expected discounted probability distribution of visiting any state <span class="math-container">$s$</span> under the policy <span class="math-container">$\pi$</span>. So now with the new joint expectation expression there's no more different step-wise <span class="math-container">$\hat A_{\pi}(s_t,a_t)$</span>, but only the unified <span class="math-container">$\hat A_{\pi}(s,a)$</span>. Therefore in terms of mathematical step, we can rewrite the RHS of the first line as <span class="math-container">$\mathbb{E}_{s \sim d_{\pi'}, a \sim \pi'}[\sum_{t=0}^\infty \gamma^t \hat A_{\pi}(s,a)]=\sum_{t=0}^\infty \gamma^t \mathbb{E}_{s \sim d_{\pi'}, a \sim \pi'}[\hat A_{\pi}(s,a)]=\frac{1}{1-\gamma}\mathbb{E}_{s \sim d_{\pi'}, a \sim \pi'}[\hat A_{\pi}(s,a)]$</span></p>
  <p>Further due to the commutative property of expectation and infinite sum under certain condition of absolutely convergent series according to <a href="https://en.wikipedia.org/wiki/Fubini%27s_theorem" rel="nofollow noreferrer">Fubini-Tonelli theorem</a> of measure theory and bear in mind that expectation is nothing but a type of weighted sum. Then it's obvious that you can use elementary geometric series formula regarding the discount factor <span class="math-container">$\gamma$</span> to get the leading coefficient.</p>
  <p>The third line of equation (11.4) is due to the idea of <a href="https://en.wikipedia.org/wiki/Importance_sampling" rel="nofollow noreferrer">importance sampling</a> since there may already be samples generated by the old policy which we want to take advantage of.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-04T18:47:50.650Z
Answers.Text: Could you please write your explanations as a step-by-step mathematical derivation?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-08T05:26:52.330Z
Answers.Text: I've added step-by-step mathy derivation as requested for the critical 2nd line, other steps seem trivial based on my described ideas. Lmk if you need additional clarification.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-04T07:07:50.277Z
Answers.LastActivityDate: 2024-09-08T05:25:39.063Z
Answers.LastEditDate: 2024-09-08T05:25:39.063Z
Answers.UserDisplayName: cinch
Body: <p>page 209, <a href="https://link.springer.com/book/10.1007/978-1-4842-9606-6" rel="nofollow noreferrer">https://link.springer.com/book/10.1007/978-1-4842-9606-6</a></p>
  <p><a href="https://i.sstatic.net/LoU0Hpdr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LoU0Hpdr.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/JuZeTK2C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JuZeTK2C.png" alt="enter image description here" /></a></p>
  <p>How is eq. (11.4) derived from eq (11.2). Could someone show step-by step derivation?</p>
  
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: Unclear derivation for the PPO method
UserDisplayName: DSPinfinity
ThreadText: # Unclear derivation for the PPO method
  
  (Asked by: username_0 on 2024-09-03)
  
  page 209, <https://link.springer.com/book/10.1007/978-1-4842-9606-6>
  
  [![enter image description here](https://i.sstatic.net/LoU0Hpdr.png)](https://i.sstatic.net/LoU0Hpdr.png)
  
  [![enter image description here](https://i.sstatic.net/JuZeTK2C.png)](https://i.sstatic.net/JuZeTK2C.png)
  
  How is eq. (11.4) derived from eq (11.2). Could someone show step-by step derivation?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-04. Score: 1 | Accepted answer
  
  The first line of equation (11.4) is copied from equation (11.2) where the definition of your book's advantage function $\hat A\_{\pi}(s\_t,a\_t)$ plays a key role and it seems you have no problem for this.
  
  The second line could be understood first rewriting the expectation wrt the random trajectory $\tau$ under the new policy $\pi'$ as the joint expectation of the random state $s$ and the random action $a$ following the distribution of $d\_{\pi'}$ and $\pi'$, respectively, since any reward trajectory is jointly decided by state and action with the environment's transition dynamics which doesn't effect advantage function at all. The author also explicitly gives the definition of $d\_{\pi}$ (thus $d\_{\pi'}$ implicitly) at the end of your quote which is the expected discounted probability distribution of visiting any state $s$ under the policy $\pi$. So now with the new joint expectation expression there's no more different step-wise $\hat A\_{\pi}(s\_t,a\_t)$, but only the unified $\hat A\_{\pi}(s,a)$. Therefore in terms of mathematical step, we can rewrite the RHS of the first line as $\mathbb{E}\_{s \sim d\_{\pi'}, a \sim \pi'}[\sum\_{t=0}^\infty \gamma^t \hat A\_{\pi}(s,a)]=\sum\_{t=0}^\infty \gamma^t \mathbb{E}\_{s \sim d\_{\pi'}, a \sim \pi'}[\hat A\_{\pi}(s,a)]=\frac{1}{1-\gamma}\mathbb{E}\_{s \sim d\_{\pi'}, a \sim \pi'}[\hat A\_{\pi}(s,a)]$
  
  Further due to the commutative property of expectation and infinite sum under certain condition of absolutely convergent series according to [Fubini-Tonelli theorem](https://en.wikipedia.org/wiki/Fubini%27s_theorem) of measure theory and bear in mind that expectation is nothing but a type of weighted sum. Then it's obvious that you can use elementary geometric series formula regarding the discount factor $\gamma$ to get the leading coefficient.
  
  The third line of equation (11.4) is due to the idea of [importance sampling](https://en.wikipedia.org/wiki/Importance_sampling) since there may already be samples generated by the old policy which we want to take advantage of.
  
  > **Comments:**
  > **username_0**: Could you please write your explanations as a step-by-step mathematical derivation?  
  > **username_1**: I've added step-by-step mathy derivation as requested for the critical 2nd line, other steps seem trivial based on my described ideas. Lmk if you need additional clarification.  
  
  ---
  
  