<p>I will refer to <span class="math-container">$\mathcal T^{\pi} $</span>as <span class="math-container">$\mathcal T$</span> and <span class="math-container">$P^{\pi}$</span> as <span class="math-container">$P$</span> for notational simplicity
<span class="math-container">\begin{align}
(\mathcal{T})^{n+1} Q &amp;= \mathcal{T}(\mathcal{T}(...(\mathcal{T}(Q))))\\
&amp;= r + \gamma P(r + \gamma P(...(r + \gamma P Q)))\\
&amp;= r + r\sum_{i=1}^{n} \gamma^i P^i + \gamma^{n+1} P^{n+1} Q
\end{align}</span></p>
<p><span class="math-container">\begin{align}
\mathcal{T}_{\lambda}Q &amp;= (1-\lambda) \sum_{n=0}^{\infty} \lambda^n (\mathcal{T})^{n+1} Q\\
&amp;=(1-\lambda)\{\lambda^0 (\mathcal T)^1Q + \lambda^1 (\mathcal T)^2Q + \lambda^2 (\mathcal T)^3Q + \ldots   \}
\end{align}</span></p>
<p>when you plug in expression for <span class="math-container">$(\mathcal T)^i Q$</span> inside this sum and rearrange you get 3 sums
<span class="math-container">\begin{equation}
\mathcal{T}_{\lambda}Q = (1-\lambda) \sum_{n=0}^{\infty} \lambda^n r + (1-\lambda)\sum_{n=1}^{\infty} \lambda^n \gamma^n P^n r + (1-\lambda)\sum_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q
\end{equation}</span></p>
<ol>
<li>sum:
<span class="math-container">\begin{equation}
(1-\lambda) \sum_{n=0}^{\infty} \lambda^n r = r
\end{equation}</span></li>
<li>sum:
<span class="math-container">\begin{equation}
(1-\lambda)\sum_{n=1}^{k} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}(1 - \lambda^k \gamma^k P^k)\lambda\gamma P r
\end{equation}</span>
As <span class="math-container">$k \rightarrow \infty$</span> and since <span class="math-container">$\gamma &lt; 1$</span> this is in the limit equal to
<span class="math-container">\begin{equation}
(1-\lambda)\sum_{n=1}^{\infty} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}\lambda\gamma P r
\end{equation}</span></li>
<li>sum:
<span class="math-container">\begin{equation}
(1-\lambda)\sum_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q = (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q 
\end{equation}</span>
If you combine all 3 you get
<span class="math-container">\begin{align}
\mathcal{T}_{\lambda}Q &amp;= r + (1-\lambda\gamma P)^{-1}\lambda\gamma P r + (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q\\
&amp;= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + \gamma PQ - \lambda\gamma PQ)\\
&amp;= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
&amp;= (1-\lambda\gamma P)^{-1}(r - \lambda \gamma P r + \lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
&amp;= (1-\lambda\gamma P)^{-1}( (\mathcal T)Q  - \lambda\gamma PQ + Q - Q)\\
&amp;= Q + (1-\lambda\gamma P)^{-1}( (\mathcal T)Q  - Q)
\end{align}</span></li>
</ol>

CC BY-SA 4.0
2021-01-21T01:43:24.663Z
Add some properties about Markov chain: https://en.wikipedia.org/wiki/Markov_chain
hijkzzz
CC BY-SA 4.0
2021-01-17T11:48:51.233Z
2021-01-17T11:48:51.233Z
Brale
<p><a href="https://i.sstatic.net/bIZrp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bIZrp.png" alt="enter image description here" /></a></p>
<p>I don't understand how the formula in the red circle is derived. The screenshot is taken from <a href="http://www.marcgbellemare.info/static/publications/harutyunyan16qlambda.pdf" rel="nofollow noreferrer">this paper</a></p>

CC BY-SA 4.0
2021-01-17T11:09:32.857Z
I recently got wondering about pretty much the same question as well. How can, for example, Q-Learning work using Auto-Differentiation when a constant is subtracted from an output of the actual Q-Network, i.e. predicted Q-Value, to compute an update of the predicted Q-Value (given that constants are commonly dropped during differentiation)? That is: why is the baseline estinate/constant, i.e. the part not being dependent on the policy network, not dropped during differentiation?
Daniel B.
<reinforcement-learning><proofs><bellman-equations><eligibility-traces>
How to prove the formula of eligibility traces operator in reinforcement learning?
hijkzzz
# How to prove the formula of eligibility traces operator in reinforcement learning?

(Asked by: username_0 on 2021-01-17)

[![enter image description here](https://i.sstatic.net/bIZrp.png)](https://i.sstatic.net/bIZrp.png)

I don't understand how the formula in the red circle is derived. The screenshot is taken from [this paper](http://www.marcgbellemare.info/static/publications/harutyunyan16qlambda.pdf)

**Tags:** reinforcement-learning, proofs, bellman-equations, eligibility-traces

> **Comments:**
> **username_1**: I recently got wondering about pretty much the same question as well. How can, for example, Q-Learning work using Auto-Differentiation when a constant is subtracted from an output of the actual Q-Network, i.e. predicted Q-Value, to compute an update of the predicted Q-Value (given that constants are commonly dropped during differentiation)? That is: why is the baseline estinate/constant, i.e. the part not being dependent on the policy network, not dropped during differentiation?  

---

## 1 answers

### Answer by username_2 on 2021-01-17. Score: 1 | Accepted answer

I will refer to $\mathcal T^{\pi} $as $\mathcal T$ and $P^{\pi}$ as $P$ for notational simplicity
\begin{align}
(\mathcal{T})^{n+1} Q &= \mathcal{T}(\mathcal{T}(...(\mathcal{T}(Q))))\\
&= r + \gamma P(r + \gamma P(...(r + \gamma P Q)))\\
&= r + r\sum\_{i=1}^{n} \gamma^i P^i + \gamma^{n+1} P^{n+1} Q
\end{align}

\begin{align}
\mathcal{T}\_{\lambda}Q &= (1-\lambda) \sum\_{n=0}^{\infty} \lambda^n (\mathcal{T})^{n+1} Q\\
&=(1-\lambda)\{\lambda^0 (\mathcal T)^1Q + \lambda^1 (\mathcal T)^2Q + \lambda^2 (\mathcal T)^3Q + \ldots \}
\end{align}

when you plug in expression for $(\mathcal T)^i Q$ inside this sum and rearrange you get 3 sums
\begin{equation}
\mathcal{T}\_{\lambda}Q = (1-\lambda) \sum\_{n=0}^{\infty} \lambda^n r + (1-\lambda)\sum\_{n=1}^{\infty} \lambda^n \gamma^n P^n r + (1-\lambda)\sum\_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q
\end{equation}

1. sum:
   \begin{equation}
   (1-\lambda) \sum\_{n=0}^{\infty} \lambda^n r = r
   \end{equation}
2. sum:
   \begin{equation}
   (1-\lambda)\sum\_{n=1}^{k} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}(1 - \lambda^k \gamma^k P^k)\lambda\gamma P r
   \end{equation}
   As $k \rightarrow \infty$ and since $\gamma < 1$ this is in the limit equal to
   \begin{equation}
   (1-\lambda)\sum\_{n=1}^{\infty} \lambda^n \gamma^n P^n r = (1-\lambda\gamma P)^{-1}\lambda\gamma P r
   \end{equation}
3. sum:
   \begin{equation}
   (1-\lambda)\sum\_{n=0}^{\infty} \lambda^n \gamma^{n+1} P^{n+1} Q = (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q
   \end{equation}
   If you combine all 3 you get
   \begin{align}
   \mathcal{T}\_{\lambda}Q &= r + (1-\lambda\gamma P)^{-1}\lambda\gamma P r + (1 - \gamma\lambda P)^{-1}(1-\lambda)\gamma P Q\\
   &= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + \gamma PQ - \lambda\gamma PQ)\\
   &= r+ (1-\lambda\gamma P)^{-1}(\lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
   &= (1-\lambda\gamma P)^{-1}(r - \lambda \gamma P r + \lambda \gamma P r + (\mathcal T)Q - r - \lambda\gamma PQ)\\
   &= (1-\lambda\gamma P)^{-1}( (\mathcal T)Q - \lambda\gamma PQ + Q - Q)\\
   &= Q + (1-\lambda\gamma P)^{-1}( (\mathcal T)Q - Q)
   \end{align}

> **Comments:**
> **username_0**: Add some properties about Markov chain: https://en.wikipedia.org/wiki/Markov_chain  

---

