Answers.Body: <p>While this may be apparent to many, I'll provide an answer anyway.</p>
  <p>In a nutshell, 2 steps:</p>
  <ol>
  <li>Bring the distribution from <span class="math-container">$\theta$</span> to <span class="math-container">$\theta '$</span> to cancel the importance ratio term. Use causality to simplify the cumulative reward term to reward-to-go. Details can be found <a href="https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work?rq=1#:%7E:text=An%20important%20thing,spinningup/extra_pg_proof1.html">here</a>, which utilized <a href="https://link.zhihu.com/?target=https%3A//spinningup.openai.com/en/latest/spinningup/rl_intro3.html%23expected-grad-log-prob-lemma" rel="nofollow noreferrer">Expected Grad-Log-Prob Lemma</a>.</li>
  <li>Bring the distribution back to <span class="math-container">$\theta$</span>. Use distribution law and the definition of marginal distribution to get the final formula.</li>
  </ol>
  <p>proof:</p>
  <p><span class="math-container">\begin{align}
  &amp;\mathop{\mathbb{E}}_{\tau\sim \theta}\left[ \left( \prod_{t=1}^{T} \frac{\pi_{\theta '}(a_t|s_t)}{\pi_{\theta }(a_t|s_t)} \right)\left(\sum_{t=1}^{T}\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t) \right) \sum_{t=1}^{T}r(s_t,a_t)\right] \\
  =&amp;\mathop{\mathbb{E}}_{\tau\sim \theta '}\left[\left(\sum_{t=1}^{T}\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\right) \sum_{t=1}^{T}r(s_t,a_t)\right] \\
  =&amp; \mathop{\mathbb{E}}_{\tau\sim \theta '}\left[\sum_{t=1}^{T}\left(\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t) \sum_{t'=t}^{T}r(s_{t'},a_{t'})\right)\right] \tag{causility} \\
  =&amp; \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta '} \left[\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right] \tag{distribution law}\\
  =&amp; \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\prod_{{t''}=1}^{T} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right]\\
  =&amp; \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\prod_{{t''}=1}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right] \tag{marginal distribution}\\
  =&amp; \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\sum_{t=1}^{T} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\sum_{t'=t}^{T} \prod_{{t''}=1}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} r(s_{t'},a_{t'}) \right]\\
  =&amp; \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\sum_{t=1}^{T} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\prod_{{t''}=1}^{t} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})}\sum_{t'=t}^{T}  r(s_{t'},a_{t'})\prod_{{t''}=t}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \right] \\
  \end{align}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-01T15:31:25.080Z
Answers.LastActivityDate: 2024-01-02T00:28:46.123Z
Answers.LastEditDate: 2024-01-02T00:28:46.123Z
Answers.UserDisplayName: zhixin
Body: <p>In <a href="https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/slides/lec-5.pdf" rel="nofollow noreferrer">lec5</a> of Sergey Levine's CS258 on RL, I can understand the derivation of the causality  simplification of on-polciy version below (page 17).</p>
  <p><a href="https://i.sstatic.net/2lmAh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2lmAh.png" alt="enter image description here" /></a></p>
  <p>But what is the detial of the derivation of <strong>the last line</strong> in the PPT below (page 25)?
  <a href="https://i.sstatic.net/l8MTt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/l8MTt.png" alt="enter image description here" /></a></p>
  <p>It seems that the cumulative reward term and the importance ratio term cannot be decompled and handled one by one.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-31T20:21:25.827Z
Comments.Text: casuality =/= causality
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-01T02:24:08.247Z
Comments.Text: @Alberto Thanks for your kind reminder. Have edited.
Comments.UserDisplayName: zhixin
Tags: <reinforcement-learning>
Title: What is the rigorous proof behind the causality simplification step of off-policy policy gradient
UserDisplayName: zhixin
ThreadText: # What is the rigorous proof behind the causality simplification step of off-policy policy gradient
  
  (Asked by: username_0 on 2023-12-31)
  
  In [lec5](https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/slides/lec-5.pdf) of Sergey Levine's CS258 on RL, I can understand the derivation of the causality simplification of on-polciy version below (page 17).
  
  [![enter image description here](https://i.sstatic.net/2lmAh.png)](https://i.sstatic.net/2lmAh.png)
  
  But what is the detial of the derivation of **the last line** in the PPT below (page 25)?
  [![enter image description here](https://i.sstatic.net/l8MTt.png)](https://i.sstatic.net/l8MTt.png)
  
  It seems that the cumulative reward term and the importance ratio term cannot be decompled and handled one by one.
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: casuality =/= causality  
  > **username_0**: @Alberto Thanks for your kind reminder. Have edited.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-01-01. Score: 1 | Accepted answer
  
  While this may be apparent to many, I'll provide an answer anyway.
  
  In a nutshell, 2 steps:
  
  1. Bring the distribution from $\theta$ to $\theta '$ to cancel the importance ratio term. Use causality to simplify the cumulative reward term to reward-to-go. Details can be found [here](https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work?rq=1#:%7E:text=An%20important%20thing,spinningup/extra_pg_proof1.html), which utilized [Expected Grad-Log-Prob Lemma](https://link.zhihu.com/?target=https%3A//spinningup.openai.com/en/latest/spinningup/rl_intro3.html%23expected-grad-log-prob-lemma).
  2. Bring the distribution back to $\theta$. Use distribution law and the definition of marginal distribution to get the final formula.
  
  proof:
  
  \begin{align}
  &\mathop{\mathbb{E}}\_{\tau\sim \theta}\left[ \left( \prod\_{t=1}^{T} \frac{\pi\_{\theta '}(a\_t|s\_t)}{\pi\_{\theta }(a\_t|s\_t)} \right)\left(\sum\_{t=1}^{T}\nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t) \right) \sum\_{t=1}^{T}r(s\_t,a\_t)\right] \\
  =&\mathop{\mathbb{E}}\_{\tau\sim \theta '}\left[\left(\sum\_{t=1}^{T}\nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t)\right) \sum\_{t=1}^{T}r(s\_t,a\_t)\right] \\
  =& \mathop{\mathbb{E}}\_{\tau\sim \theta '}\left[\sum\_{t=1}^{T}\left(\nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t) \sum\_{t'=t}^{T}r(s\_{t'},a\_{t'})\right)\right] \tag{causility} \\
  =& \sum\_{t=1}^{T} \sum\_{t'=t}^{T} \mathop{\mathbb{E}}\_{\tau\sim \theta '} \left[\nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t)r(s\_{t'},a\_{t'}) \right] \tag{distribution law}\\
  =& \sum\_{t=1}^{T} \sum\_{t'=t}^{T} \mathop{\mathbb{E}}\_{\tau\sim \theta} \left[\prod\_{{t''}=1}^{T} \frac{\pi\_{\theta '}(a\_{t''}|s\_{t''})}{\pi\_{\theta }(a\_{t''}|s\_{t''})} \nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t)r(s\_{t'},a\_{t'}) \right]\\
  =& \sum\_{t=1}^{T} \sum\_{t'=t}^{T} \mathop{\mathbb{E}}\_{\tau\sim \theta} \left[\prod\_{{t''}=1}^{t'} \frac{\pi\_{\theta '}(a\_{t''}|s\_{t''})}{\pi\_{\theta }(a\_{t''}|s\_{t''})} \nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t)r(s\_{t'},a\_{t'}) \right] \tag{marginal distribution}\\
  =& \mathop{\mathbb{E}}\_{\tau\sim \theta} \left[\sum\_{t=1}^{T} \nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t)\sum\_{t'=t}^{T} \prod\_{{t''}=1}^{t'} \frac{\pi\_{\theta '}(a\_{t''}|s\_{t''})}{\pi\_{\theta }(a\_{t''}|s\_{t''})} r(s\_{t'},a\_{t'}) \right]\\
  =& \mathop{\mathbb{E}}\_{\tau\sim \theta} \left[\sum\_{t=1}^{T} \nabla\_{\theta'}\log \pi\_{\theta '}(a\_t|s\_t)\prod\_{{t''}=1}^{t} \frac{\pi\_{\theta '}(a\_{t''}|s\_{t''})}{\pi\_{\theta }(a\_{t''}|s\_{t''})}\sum\_{t'=t}^{T} r(s\_{t'},a\_{t'})\prod\_{{t''}=t}^{t'} \frac{\pi\_{\theta '}(a\_{t''}|s\_{t''})}{\pi\_{\theta }(a\_{t''}|s\_{t''})} \right] \\
  \end{align}
  
  
  ---
  
  