Answers.Body: <p>Let's assume <span class="math-container">$\gamma = 1$</span> to simplify things
  <span class="math-container">\begin{align}
  \mathbb E_{\tau|\pi} [\sum_{t = 0}^{\infty} A_\pi(s_t, a_t)] &amp;= \mathbb E_{\tau|\pi}[A_\pi(s_0, a_0) + \ldots A_\pi(s_i, a_i) + \ldots]\\
  &amp;= \mathbb E_{a_0 \sim \pi,s_1 \sim P(s_1|s_0, a_0)}[A_\pi(s_0, a_0)] + \ldots + \mathbb E_{a_i \sim \pi,s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[A_\pi(s_i, a_i)] + \ldots
  \end{align}</span>
  if we observe only <span class="math-container">$i$</span>-th timestep
  <span class="math-container">\begin{align}
  \mathbb E_{a_i \sim \pi,s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[A_\pi(s_i, a_i)] &amp;= \sum_{a'} (\mathbb E_{s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[A_\pi(s_i, a')]) \pi(a'|s_i)\\
  &amp;= \sum_{a'} (\mathbb E_{s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[\mathbb E_{s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[r(s_i) + \\ &amp;V_\pi(s_{i+1}) - V_\pi(s)]]) \pi(a'|s_i)
  \end{align}</span></p>
  <p><span class="math-container">\begin{equation}
  \mathbb E[\mathbb E[f]] = \mathbb E[f]
  \end{equation}</span></p>
  <p><span class="math-container">\begin{align}
  \mathbb E_{a_i \sim \pi,s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[A_\pi(s_i, a_i)] &amp;= \sum_{a'} (\mathbb E_{s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[r(s_i) + V_\pi(s_{i+1}) - V_\pi(s)]) \pi(a'|s_i)\\
  &amp;= \mathbb E_{a_i \sim \pi,s_{i+1} \sim P(s_{i+1}|s_i, a_i)}[r(s_i) + V_\pi(s_{i+1}) - V_\pi(s)]
  \end{align}</span></p>
  <p>now for all timesteps sum everything up.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-22T10:18:21.273Z
Answers.LastActivityDate: 2023-01-21T17:34:36.520Z
Answers.LastEditDate: 2023-01-21T17:34:36.520Z
Answers.UserDisplayName: Brale
Body: <p>In the <a href="https://arxiv.org/pdf/1502.05477.pdf" rel="nofollow noreferrer">Trust Region Policy Optimization</a> paper, in Lemma 1 of Appendix A, I didn't quite understand the transition from (21) from (20). In going from (20) to (21), <span class="math-container">$A^\pi(s_t, a_t)$</span> is substituted with its value. The value of <span class="math-container">$A^\pi(s_t, a_t)$</span> is given as <span class="math-container">$\mathbb{E}_{s'∼P(s'|s,a)}[r(s) + \gamma V_\pi(s') − V_\pi(s)]$</span> at the very beginning of the proof. But when <span class="math-container">$A^\pi(s_t, a_t)$</span> gets substituted, I don't see the expectation (over <span class="math-container">$s'∼P(s'|s,a)$</span>) appearing anywhere. It will be of great help if somebody lends some light on this.</p>
  
Tags: <reinforcement-learning><proofs><papers><trust-region-policy-optimization>
Title: In lemma 1 of the TRPO paper, why isn't the expectation over $s'∼P(s'|s,a)$?
UserDisplayName: A Das
ThreadText: # In lemma 1 of the TRPO paper, why isn't the expectation over $s'∼P(s'|s,a)$?
  
  (Asked by: username_0 on 2019-11-21)
  
  In the [Trust Region Policy Optimization](https://arxiv.org/pdf/1502.05477.pdf) paper, in Lemma 1 of Appendix A, I didn't quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s\_t, a\_t)$ is substituted with its value. The value of $A^\pi(s\_t, a\_t)$ is given as $\mathbb{E}\_{s'∼P(s'|s,a)}[r(s) + \gamma V\_\pi(s') − V\_\pi(s)]$ at the very beginning of the proof. But when $A^\pi(s\_t, a\_t)$ gets substituted, I don't see the expectation (over $s'∼P(s'|s,a)$) appearing anywhere. It will be of great help if somebody lends some light on this.
  
  **Tags:** reinforcement-learning, proofs, papers, trust-region-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-22. Score: 0
  
  Let's assume $\gamma = 1$ to simplify things
  \begin{align}
  \mathbb E\_{\tau|\pi} [\sum\_{t = 0}^{\infty} A\_\pi(s\_t, a\_t)] &= \mathbb E\_{\tau|\pi}[A\_\pi(s\_0, a\_0) + \ldots A\_\pi(s\_i, a\_i) + \ldots]\\
  &= \mathbb E\_{a\_0 \sim \pi,s\_1 \sim P(s\_1|s\_0, a\_0)}[A\_\pi(s\_0, a\_0)] + \ldots + \mathbb E\_{a\_i \sim \pi,s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[A\_\pi(s\_i, a\_i)] + \ldots
  \end{align}
  if we observe only $i$-th timestep
  \begin{align}
  \mathbb E\_{a\_i \sim \pi,s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[A\_\pi(s\_i, a\_i)] &= \sum\_{a'} (\mathbb E\_{s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[A\_\pi(s\_i, a')]) \pi(a'|s\_i)\\
  &= \sum\_{a'} (\mathbb E\_{s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[\mathbb E\_{s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[r(s\_i) + \\ &V\_\pi(s\_{i+1}) - V\_\pi(s)]]) \pi(a'|s\_i)
  \end{align}
  
  \begin{equation}
  \mathbb E[\mathbb E[f]] = \mathbb E[f]
  \end{equation}
  
  \begin{align}
  \mathbb E\_{a\_i \sim \pi,s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[A\_\pi(s\_i, a\_i)] &= \sum\_{a'} (\mathbb E\_{s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[r(s\_i) + V\_\pi(s\_{i+1}) - V\_\pi(s)]) \pi(a'|s\_i)\\
  &= \mathbb E\_{a\_i \sim \pi,s\_{i+1} \sim P(s\_{i+1}|s\_i, a\_i)}[r(s\_i) + V\_\pi(s\_{i+1}) - V\_\pi(s)]
  \end{align}
  
  now for all timesteps sum everything up.
  
  
  ---
  
  