Answers.Body: <p>There's not much to derive here it's simply a definition of Bellman operator, it comes from Bellman equation. If you're wondering why
  <span class="math-container">\begin{equation}
  Q^{\pi} = (I - \gamma P^{\pi})^{-1}r \tag{1}
  \end{equation}</span>
  they state that <span class="math-container">$Q^{\pi}$</span> is a fixed point which means if you apply Bellman operator to it you get the same value
  <span class="math-container">\begin{equation}
  T^{\pi}(Q^{\pi}) = Q^{\pi}
  \end{equation}</span>
  You can easily check that since from <span class="math-container">$(1)$</span>
  <span class="math-container">\begin{equation}
  r = (I-\gamma P^{\pi})Q^{\pi}
  \end{equation}</span>
  if you plug it in definition of Bellman operator you get
  <span class="math-container">\begin{align}
  T^{\pi}(Q^{\pi}) &amp;= r + \gamma P^{\pi} Q^{\pi}\\
  &amp;= (I - \gamma P^{\pi})Q^{\pi}+  \gamma P^{\pi} Q^{\pi}\\
  &amp;= Q^{\pi}
  \end{align}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-24T18:23:15.337Z
Answers.Text: In other words, this proof shows that, if you have some value function $Q^\pi$, such that $r = (I - \gamma P^\pi)Q^\pi$, then $Q^\pi$ is a fixed-point of $T^\pi$. So, to actually answer the question: $Q^{\pi} = (I - \gamma P^{\pi})^{-1}r$ is true because that's what makes $Q^{\pi}$ a fixed point. Note that you could have just used $Q$ (rather than $Q^\pi$) in this derivation, i.e. any value function $Q$ such that $r = (I - \gamma P^\pi)Q$ is a fixed-point.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T15:44:55.420Z
Answers.Text: What is the exact meaning of the matrix notation in this context? I ask this in particular because the operator $P^{\pi}$ is defined as
  \begin{equation}
  (P^{\pi}Q)(s, a) = \sum_{s'}P(s'\vert s, a)\sum_{a'}\pi(a'\vert s') Q(s', a')
  \end{equation}
  which makes $P^{\pi}Q$ not being a common matrix-matrix multiplication
Answers.UserDisplayName: Federico Taschin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T16:28:33.013Z
Answers.Text: In particular, isn't here $P^{\pi}$ a tensor? When we use the Bellman operators on the value function we are marginalizing out the actions, but here we apply it explicitly to $Q(s, a)$ for each action, thus $P^{\pi}$ needs to be an $S\times S \times A$ tensor?
Answers.UserDisplayName: Federico Taschin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T18:05:59.593Z
Answers.Text: David Silver [lecture](https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf) (especially slides 22 onwards) can give you more insight into matrix equation, or you can just ask another question on website. $P$ could be a tensor or it could be a matrix of dimensions $(S \times A) \times S$ with appropriate reshaping.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-24T17:54:35.217Z
Answers.LastActivityDate: 2020-12-24T18:07:51.773Z
Answers.LastEditDate: 2020-12-24T18:07:51.773Z
Answers.UserDisplayName: Brale
Body: <p>Reading the Retrace paper (<a href="https://arxiv.org/pdf/1606.02647.pdf" rel="nofollow noreferrer">Safe and efficient off-policy reinforcement learning</a>) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive those forms? Could you point me to some reference in which the matter is explained?</p>
  <p>I am familiar with the tabular RL framework, but I'm having trouble understanding the steps from operators to this matrix form. For example, why does <span class="math-container">$Q^{\pi} = (I -\gamma P^{\pi})^{-1}r$</span>? I know that for the value <span class="math-container">$V$</span> we can write
  <span class="math-container">\begin{align}
  V =  R + \gamma P^{\pi} V \\
  V - \gamma P^{\pi} V =  R \\
  V (I -\gamma P^{\pi}) =  R \\ 
  V = R(I - \gamma P^{\pi})^{-1}
  \end{align}</span>
  but this seems slightly different.</p>
  <p>Picture from <em>Safe and efficient off-policy reinforcement learning</em>:</p>
  <p><a href="https://i.sstatic.net/hXzw6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hXzw6.png" alt="Picture from Retrace paper" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-24T12:01:47.553Z
Comments.Text: [Here](https://ai.stackexchange.com/q/11057/2444) you have a related but more general question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><reference-request><value-functions><bellman-equations><bellman-operators>
Title: How to derive matrix form of the Bellman operators?
UserDisplayName: Federico Taschin
ThreadText: # How to derive matrix form of the Bellman operators?
  
  (Asked by: username_0 on 2020-12-24)
  
  Reading the Retrace paper ([Safe and efficient off-policy reinforcement learning](https://arxiv.org/pdf/1606.02647.pdf)) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive those forms? Could you point me to some reference in which the matter is explained?
  
  I am familiar with the tabular RL framework, but I'm having trouble understanding the steps from operators to this matrix form. For example, why does $Q^{\pi} = (I -\gamma P^{\pi})^{-1}r$? I know that for the value $V$ we can write
  \begin{align}
  V = R + \gamma P^{\pi} V \\
  V - \gamma P^{\pi} V = R \\
  V (I -\gamma P^{\pi}) = R \\
  V = R(I - \gamma P^{\pi})^{-1}
  \end{align}
  but this seems slightly different.
  
  Picture from *Safe and efficient off-policy reinforcement learning*:
  
  [![Picture from Retrace paper](https://i.sstatic.net/hXzw6.png)](https://i.sstatic.net/hXzw6.png)
  
  **Tags:** reinforcement-learning, reference-request, value-functions, bellman-equations, bellman-operators
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/11057/2444) you have a related but more general question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-24. Score: 2 | Accepted answer
  
  There's not much to derive here it's simply a definition of Bellman operator, it comes from Bellman equation. If you're wondering why
  \begin{equation}
  Q^{\pi} = (I - \gamma P^{\pi})^{-1}r \tag{1}
  \end{equation}
  they state that $Q^{\pi}$ is a fixed point which means if you apply Bellman operator to it you get the same value
  \begin{equation}
  T^{\pi}(Q^{\pi}) = Q^{\pi}
  \end{equation}
  You can easily check that since from $(1)$
  \begin{equation}
  r = (I-\gamma P^{\pi})Q^{\pi}
  \end{equation}
  if you plug it in definition of Bellman operator you get
  \begin{align}
  T^{\pi}(Q^{\pi}) &= r + \gamma P^{\pi} Q^{\pi}\\
  &= (I - \gamma P^{\pi})Q^{\pi}+ \gamma P^{\pi} Q^{\pi}\\
  &= Q^{\pi}
  \end{align}
  
  > **Comments:**
  > **username_1**: In other words, this proof shows that, if you have some value function $Q^\pi$, such that $r = (I - \gamma P^\pi)Q^\pi$, then $Q^\pi$ is a fixed-point of $T^\pi$. So, to actually answer the question: $Q^{\pi} = (I - \gamma P^{\pi})^{-1}r$ is true because that's what makes $Q^{\pi}$ a fixed point. Note that you could have just used $Q$ (rather than $Q^\pi$) in this derivation, i.e. any value function $Q$ such that $r = (I - \gamma P^\pi)Q$ is a fixed-point.  
  > **username_0**: What is the exact meaning of the matrix notation in this context? I ask this in particular because the operator $P^{\pi}$ is defined as \begin{equation} (P^{\pi}Q)(s, a) = \sum_{s'}P(s'\vert s, a)\sum_{a'}\pi(a'\vert s') Q(s', a') \end{equation} which makes $P^{\pi}Q$ not being a common matrix-matrix multiplication  
  > **username_0**: In particular, isn't here $P^{\pi}$ a tensor? When we use the Bellman operators on the value function we are marginalizing out the actions, but here we apply it explicitly to $Q(s, a)$ for each action, thus $P^{\pi}$ needs to be an $S\times S \times A$ tensor?  
  > **username_2**: David Silver [lecture](https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf) (especially slides 22 onwards) can give you more insight into matrix equation, or you can just ask another question on website. $P$ could be a tensor or it could be a matrix of dimensions $(S \times A) \times S$ with appropriate reshaping.  
  
  ---
  
  