Answers.Body: <p>I found the answer in <a href="http://chercheurs.lille.inria.fr/%7Emunos/papers/files/lspi_chapter.pdf" rel="nofollow noreferrer">this paper</a></p>
  <p><a href="https://i.sstatic.net/cwYcJWKg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cwYcJWKg.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-12T18:13:15.727Z
Answers.LastActivityDate: 2024-09-12T18:13:15.727Z
Answers.UserDisplayName: Simon
Answers.Body: <p>As far as I understand it, the policy <span class="math-container">$\Pi$</span> is a map of a state to an action when in that state. If the action and state space are both discrete with <span class="math-container">$N$</span> available actions and <span class="math-container">$M$</span> available states then using the labels <span class="math-container">$s = 1,2,3, ... M$</span> for state and <span class="math-container">$a = 1,2,3, ... N$</span> for action and write the components of <span class="math-container">$Q^\pi$</span> out in the form of <span class="math-container">$Q_{sa}$</span></p>
  <p><span class="math-container">$$ Q^\pi = \left ( \begin{matrix}Q_{11}\\ Q_{12}\\ ...\\ Q_{1N}\\ Q_{21}\\ Q_{22}\\ ...\\ Q_{2N}\\ ...\\ Q_{M1}\\ Q_{M2}\\ ...\\ Q_{MN}\\ \end{matrix} \right ) 
  $$</span>
  then a non-deterministic policy with probability of <span class="math-container">$p_{sa}$</span> to apply an action <span class="math-container">$a$</span> when in state <span class="math-container">$s$</span> could be written as
  <span class="math-container">$$
  \Pi = \left ( 
  \begin{matrix}
  p_{11} &amp; p_{12} &amp; \ldots &amp; p_{1N} &amp; 0      &amp; 0      &amp; \ldots &amp; 0      &amp; 0 &amp; 0 &amp; 0  &amp; \ldots &amp; \ldots \cr
  0      &amp; 0      &amp; \ldots &amp; 0      &amp; p_{21} &amp; p_{22} &amp; \ldots &amp; p_{2N} &amp; 0 &amp; 0 &amp; 0  &amp; \ldots &amp; \ldots \cr  
  \vdots &amp; \vdots &amp; \ddots &amp; 0      &amp; 0      &amp; 0      &amp; \ldots &amp; 0      &amp; p_{31} &amp; p_{32} &amp; \ldots \cr
  \vdots \cr
  \vdots \cr
  0      &amp; 0      &amp; 0      &amp; 0      &amp; 0      &amp; 0      &amp; \ldots &amp; 0      &amp; 0 &amp; 0 &amp; 0 &amp; \ldots &amp; \ldots &amp; p_{M1} &amp; p_{M2} &amp; \ldots &amp; p_{MN} \cr 
  \end{matrix} 
  \right ) 
  $$</span>
  where <span class="math-container">$\Sigma_a p_{sa} = 1$</span>. The <span class="math-container">$\Pi$</span> is a sort of <span class="math-container">$(1 \times N)$</span>-block diagonal matrix.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-13T23:08:23.807Z
Answers.LastActivityDate: 2024-09-13T23:08:23.807Z
Answers.UserDisplayName: stackoverblown
Body: <p>It's rather easy to write construct the matrices for the matrix form of the Bellman equation for the V-function (they are everywhere on the internet) but I can't find the equivalent version for the Q-function. The only one I found is from <a href="https://www.jmlr.org/papers/v4/lagoudakis03a.html" rel="nofollow noreferrer">the LSTD paper</a>, but I don't understand how to construct the matrix <span class="math-container">$\Pi_{\pi}$</span>. Usually, <span class="math-container">$\Pi$</span> has shape <span class="math-container">$|S| \times |A|$</span>, but <span class="math-container">$\Pi_{\pi}$</span> has shape <span class="math-container">$|S| \times |S||A|$</span>. What exactly is it?
  Is there any other source of information on this?</p>
  <p><a href="https://i.sstatic.net/bZn0Iy8U.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bZn0Iy8U.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><bellman-equations><policy-iteration>
Title: Bellman equation matrix form for Q-function
UserDisplayName: Simon
ThreadText: # Bellman equation matrix form for Q-function
  
  (Asked by: username_0 on 2024-09-12)
  
  It's rather easy to write construct the matrices for the matrix form of the Bellman equation for the V-function (they are everywhere on the internet) but I can't find the equivalent version for the Q-function. The only one I found is from [the LSTD paper](https://www.jmlr.org/papers/v4/lagoudakis03a.html), but I don't understand how to construct the matrix $\Pi\_{\pi}$. Usually, $\Pi$ has shape $|S| \times |A|$, but $\Pi\_{\pi}$ has shape $|S| \times |S||A|$. What exactly is it?
  Is there any other source of information on this?
  
  [![enter image description here](https://i.sstatic.net/bZn0Iy8U.png)](https://i.sstatic.net/bZn0Iy8U.png)
  
  **Tags:** reinforcement-learning, bellman-equations, policy-iteration
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2024-09-12. Score: 1 | Accepted answer
  
  I found the answer in [this paper](http://chercheurs.lille.inria.fr/%7Emunos/papers/files/lspi_chapter.pdf)
  
  [![enter image description here](https://i.sstatic.net/cwYcJWKg.png)](https://i.sstatic.net/cwYcJWKg.png)
  
  
  ---
  
  ### Answer by username_1 on 2024-09-13. Score: 1
  
  As far as I understand it, the policy $\Pi$ is a map of a state to an action when in that state. If the action and state space are both discrete with $N$ available actions and $M$ available states then using the labels $s = 1,2,3, ... M$ for state and $a = 1,2,3, ... N$ for action and write the components of $Q^\pi$ out in the form of $Q\_{sa}$
  
  $$ Q^\pi = \left ( \begin{matrix}Q\_{11}\\ Q\_{12}\\ ...\\ Q\_{1N}\\ Q\_{21}\\ Q\_{22}\\ ...\\ Q\_{2N}\\ ...\\ Q\_{M1}\\ Q\_{M2}\\ ...\\ Q\_{MN}\\ \end{matrix} \right )
  $$
  then a non-deterministic policy with probability of $p\_{sa}$ to apply an action $a$ when in state $s$ could be written as
  $$
  \Pi = \left (
  \begin{matrix}
  p\_{11} & p\_{12} & \ldots & p\_{1N} & 0 & 0 & \ldots & 0 & 0 & 0 & 0 & \ldots & \ldots \cr
  0 & 0 & \ldots & 0 & p\_{21} & p\_{22} & \ldots & p\_{2N} & 0 & 0 & 0 & \ldots & \ldots \cr
  \vdots & \vdots & \ddots & 0 & 0 & 0 & \ldots & 0 & p\_{31} & p\_{32} & \ldots \cr
  \vdots \cr
  \vdots \cr
  0 & 0 & 0 & 0 & 0 & 0 & \ldots & 0 & 0 & 0 & 0 & \ldots & \ldots & p\_{M1} & p\_{M2} & \ldots & p\_{MN} \cr
  \end{matrix}
  \right )
  $$
  where $\Sigma\_a p\_{sa} = 1$. The $\Pi$ is a sort of $(1 \times N)$-block diagonal matrix.
  
  
  ---
  
  