Answers.Body: <p>I will refer to <span class="math-container">$\mathcal T^{\pi} $</span>as <span class="math-container">$\mathcal T$</span> and <span class="math-container">$P^{\pi}$</span> as <span class="math-container">$P$</span> for notational simplicity
  <span class="math-container">\begin{align}
  (\mathcal{T})^{n+1} Q &amp;= \mathcal{T}(\mathcal{T}(...(\mathcal{T}(Q))))\\
  &amp;= r + \gamma P(r + \gamma P(...(r + \gamma P Q)))\\
  &amp;= r + r\sum_{i=1}^{n} \gamma^i P^i + \gamma^{n+1} P^{n+1} Q
  \end{align}</span></p>
  <p><span class="math-container">\begin{align}
  \mathcal{T}_{\lambda}Q &amp;= (1-\lambda) \sum_{n=0}^{\infty} \lambda^n (\mathcal{T})^{n+1} Q\\
  &amp;=(1-\lambda)\{\lambda^0 (\mathcal T)^1Q + \lambda^1 (\mathcal T)^2Q + \lambda^2 (\mathcal T)^3Q + \ldots   \}
  \end{align}</span></p>
  <p>when you plug in expression for <span class="math-container">$(\mathcal T)^i Q$</span> inside this sum and rearrange you get 3 sums
  <span class="math-container">\begin{equation}
  \mathcal{T}_{\lambda}Q = (1-\lambda) \sum_{n=0}^{\infty} \lambda^n r + (1-\lambda)\sum_{n=1}^{\infty} \lambda^n \gamma^n P^n r + (1-\lambda)\sum_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q
  \end{equation}</span></p>
  <ol>
  <li>sum:
  <span class="math-container">\begin{equation}
  (1-\lambda) \sum_{n=0}^{\infty} \lambda^n r = r
  \end{equation}</span></li>
  <li>sum:
  <span class="math-container">\begin{equation}
  (1-\lambda)\sum_{n=1}^{k} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}(1 - \lambda^k \gamma^k P^k)\lambda\gamma P r
  \end{equation}</span>
  As <span class="math-container">$k \rightarrow \infty$</span> and since <span class="math-container">$\gamma &lt; 1$</span> this is in the limit equal to
  <span class="math-container">\begin{equation}
  (1-\lambda)\sum_{n=1}^{\infty} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}\lambda\gamma P r
  \end{equation}</span></li>
  <li>sum:
  <span class="math-container">\begin{equation}
  (1-\lambda)\sum_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q = (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q 
  \end{equation}</span>
  If you combine all 3 you get
  <span class="math-container">\begin{align}
  \mathcal{T}_{\lambda}Q &amp;= r + (1-\lambda\gamma P)^{-1}\lambda\gamma P r + (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q\\
  &amp;= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + \gamma PQ - \lambda\gamma PQ)\\
  &amp;= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
  &amp;= (1-\lambda\gamma P)^{-1}(r - \lambda \gamma P r + \lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
  &amp;= (1-\lambda\gamma P)^{-1}( (\mathcal T)Q  - \lambda\gamma PQ + Q - Q)\\
  &amp;= Q + (1-\lambda\gamma P)^{-1}( (\mathcal T)Q  - Q)
  \end{align}</span></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T01:43:24.663Z
Answers.Text: Add some properties about Markov chain: https://en.wikipedia.org/wiki/Markov_chain
Answers.UserDisplayName: hijkzzz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T11:48:51.233Z
Answers.LastActivityDate: 2021-01-17T11:48:51.233Z
Answers.UserDisplayName: Brale
Body: <p><a href="https://i.sstatic.net/bIZrp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bIZrp.png" alt="enter image description here" /></a></p>
  <p>I don't understand how the formula in the red circle is derived. The screenshot is taken from <a href="http://www.marcgbellemare.info/static/publications/harutyunyan16qlambda.pdf" rel="nofollow noreferrer">this paper</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T11:09:32.857Z
Comments.Text: I recently got wondering about pretty much the same question as well. How can, for example, Q-Learning work using Auto-Differentiation when a constant is subtracted from an output of the actual Q-Network, i.e. predicted Q-Value, to compute an update of the predicted Q-Value (given that constants are commonly dropped during differentiation)? That is: why is the baseline estinate/constant, i.e. the part not being dependent on the policy network, not dropped during differentiation?
Comments.UserDisplayName: Daniel B.
Tags: <reinforcement-learning><proofs><bellman-equations><eligibility-traces>
Title: How to prove the formula of eligibility traces operator in reinforcement learning?
UserDisplayName: hijkzzz
ThreadText: # How to prove the formula of eligibility traces operator in reinforcement learning?
  
  (Asked by: username_0 on 2021-01-17)
  
  [![enter image description here](https://i.sstatic.net/bIZrp.png)](https://i.sstatic.net/bIZrp.png)
  
  I don't understand how the formula in the red circle is derived. The screenshot is taken from [this paper](http://www.marcgbellemare.info/static/publications/harutyunyan16qlambda.pdf)
  
  **Tags:** reinforcement-learning, proofs, bellman-equations, eligibility-traces
  > **Comments:**
  > **username_1**: I recently got wondering about pretty much the same question as well. How can, for example, Q-Learning work using Auto-Differentiation when a constant is subtracted from an output of the actual Q-Network, i.e. predicted Q-Value, to compute an update of the predicted Q-Value (given that constants are commonly dropped during differentiation)? That is: why is the baseline estinate/constant, i.e. the part not being dependent on the policy network, not dropped during differentiation?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-17. Score: 1 | Accepted answer
  
  I will refer to $\mathcal T^{\pi} $as $\mathcal T$ and $P^{\pi}$ as $P$ for notational simplicity
  \begin{align}
  (\mathcal{T})^{n+1} Q &= \mathcal{T}(\mathcal{T}(...(\mathcal{T}(Q))))\\
  &= r + \gamma P(r + \gamma P(...(r + \gamma P Q)))\\
  &= r + r\sum\_{i=1}^{n} \gamma^i P^i + \gamma^{n+1} P^{n+1} Q
  \end{align}
  
  \begin{align}
  \mathcal{T}\_{\lambda}Q &= (1-\lambda) \sum\_{n=0}^{\infty} \lambda^n (\mathcal{T})^{n+1} Q\\
  &=(1-\lambda)\{\lambda^0 (\mathcal T)^1Q + \lambda^1 (\mathcal T)^2Q + \lambda^2 (\mathcal T)^3Q + \ldots \}
  \end{align}
  
  when you plug in expression for $(\mathcal T)^i Q$ inside this sum and rearrange you get 3 sums
  \begin{equation}
  \mathcal{T}\_{\lambda}Q = (1-\lambda) \sum\_{n=0}^{\infty} \lambda^n r + (1-\lambda)\sum\_{n=1}^{\infty} \lambda^n \gamma^n P^n r + (1-\lambda)\sum\_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q
  \end{equation}
  
  1. sum:
     \begin{equation}
     (1-\lambda) \sum\_{n=0}^{\infty} \lambda^n r = r
     \end{equation}
  2. sum:
     \begin{equation}
     (1-\lambda)\sum\_{n=1}^{k} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}(1 - \lambda^k \gamma^k P^k)\lambda\gamma P r
     \end{equation}
     As $k \rightarrow \infty$ and since $\gamma < 1$ this is in the limit equal to
     \begin{equation}
     (1-\lambda)\sum\_{n=1}^{\infty} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}\lambda\gamma P r
     \end{equation}
  3. sum:
     \begin{equation}
     (1-\lambda)\sum\_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q = (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q
     \end{equation}
     If you combine all 3 you get
     \begin{align}
     \mathcal{T}\_{\lambda}Q &= r + (1-\lambda\gamma P)^{-1}\lambda\gamma P r + (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q\\
     &= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + \gamma PQ - \lambda\gamma PQ)\\
     &= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
     &= (1-\lambda\gamma P)^{-1}(r - \lambda \gamma P r + \lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
     &= (1-\lambda\gamma P)^{-1}( (\mathcal T)Q - \lambda\gamma PQ + Q - Q)\\
     &= Q + (1-\lambda\gamma P)^{-1}( (\mathcal T)Q - Q)
     \end{align}
  
  > **Comments:**
  > **username_0**: Add some properties about Markov chain: https://en.wikipedia.org/wiki/Markov_chain  
  
  ---
  
  