Answers.Body: <p>Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. Now, in your equation (2) you have conditioned on taking action <span class="math-container">$a'$</span> in the inner expectation - this is not what happens in the state-action value function, you <em>do not</em> condition on knowing <span class="math-container">$A_{t+1}$</span>, it is chosen according to the policy <span class="math-container">$\pi$</span> as per the definition of a bellman equation.</p>
  <p>If you want to see a 'recursion' between state action value functions, note that</p>
  <p><span class="math-container">$$v_\pi(s) = \sum_a \pi(a|s)q_\pi(s,a)\;,$$</span></p>
  <p>Your equation (5) is incorrect -- you need to drop the outter sum over <span class="math-container">$a$</span> as we have conditioned on knowing <span class="math-container">$a$</span>. I will drop the <span class="math-container">$\pi$</span> subscripts for ease on notation, and we can see a 'recursion' for state-action value functions as:</p>
  <p><span class="math-container">$$q(s,a) = \sum_{s',r}p(s',r|s,a)\left(r + \gamma \left[\sum_{a'} \pi(a'|s')q(s',a')\right]\right)\;.$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T10:28:28.830Z
Answers.LastActivityDate: 2020-06-17T11:43:04.100Z
Answers.LastEditDate: 2020-06-17T11:43:04.100Z
Answers.UserDisplayName: David
Body: <p>When deriving the Bellman equation for <span class="math-container">$q_\pi(s,a)$</span>, we have</p>
  <p><span class="math-container">$q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$</span> (1)</p>
  <p>This is what is confusing me, at this point, for the Bellman equation for <span class="math-container">$q_\pi(s,a)$</span>, we write <span class="math-container">$G_{t+1}$</span> as an expected value, conditioned on <span class="math-container">$s'$</span> <em>and</em> <span class="math-container">$a'$</span> of the <em>action value function</em> at <span class="math-container">$s'$</span>, otherwise, there is no recursion with respect to <span class="math-container">$q_\pi(s,a)$</span>, and therefore no Bellman equation. Namely,</p>
  <p><span class="math-container">$ = \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma E_\pi[G_{t+1}|S_{t+1} = s', A_{t+1} = a'])$</span> (2)</p>
  <p>which introduces the recursion of <span class="math-container">$q$</span>,</p>
  <p><span class="math-container">$ = \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma q_\pi(s',a'))$</span> (3)</p>
  <p>which should be the Bellman equation for <span class="math-container">$q_\pi(s,a)$</span>, right?</p>
  <p>On the other hand, when connecting <span class="math-container">$q_\pi(s,a)$</span> with <span class="math-container">$v_\pi(s')$</span>, <a href="https://ai.stackexchange.com/questions/21684/why-state-action-value-function-as-an-expected-value-of-the-return-and-state-val/21687#21687">in this answer</a>, I believe this is done</p>
  <p><span class="math-container">$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma E_{\pi}[G_{t+1} | S_{t+1} = s'])$</span> (4)</p>
  <p><span class="math-container">$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma v_\pi(s'))$</span> (5)</p>
  <p>Is the difference between using the expectation <span class="math-container">$E_{\pi}[G_{t+1} | S_{t+1} = s', A_{t+1} = a']$</span> in (3) and the expectation <span class="math-container">$E_{\pi}[G_{t+1} | S_{t+1} = s']$</span> in <span class="math-container">$(4)$</span> simply the difference in how we choose to express the expected return <span class="math-container">$G_{t+1}$</span> at <span class="math-container">$s'$</span> in the definition of <span class="math-container">$q_\pi(s,a)$</span>?</p>
  <p>In <span class="math-container">$3$</span>, we express the total return at <span class="math-container">$s'$</span> using the action value function</p>
  <p><a href="https://i.sstatic.net/D8jyk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/D8jyk.png" alt="enter image description here" /></a></p>
  <p>leading to the recursion and the Bellman equation, and in <span class="math-container">$4$</span>, the total return is expressed at <span class="math-container">$s'$</span> using the value function</p>
  <p><a href="https://i.sstatic.net/rlxHz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rlxHz.png" alt="enter image description here" /></a></p>
  <p>leading to <span class="math-container">$q_\pi(s,a) = q_\pi(s,a,v_\pi(s'))$</span>?</p>
  
Tags: <reinforcement-learning><value-functions><bellman-equations>
Title: Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$
UserDisplayName: tmaric
ThreadText: # Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$
  
  (Asked by: username_0 on 2020-06-17)
  
  When deriving the Bellman equation for $q\_\pi(s,a)$, we have
  
  $q\_\pi(s,a) = E\_\pi[G\_t | S\_t = s, A\_t = a] = E\_\pi[R\_{t+1} + \gamma G\_{t+1} | S\_t = s, A\_t = a]$ (1)
  
  This is what is confusing me, at this point, for the Bellman equation for $q\_\pi(s,a)$, we write $G\_{t+1}$ as an expected value, conditioned on $s'$ *and* $a'$ of the *action value function* at $s'$, otherwise, there is no recursion with respect to $q\_\pi(s,a)$, and therefore no Bellman equation. Namely,
  
  $ = \sum\_{a \in A} \pi(a |s) \sum\_{s' \in S} \sum\_{r \in R} p(s',r|s,a)(r + \gamma E\_\pi[G\_{t+1}|S\_{t+1} = s', A\_{t+1} = a'])$ (2)
  
  which introduces the recursion of $q$,
  
  $ = \sum\_{a \in A} \pi(a |s) \sum\_{s' \in S} \sum\_{r \in R} p(s',r|s,a)(r + \gamma q\_\pi(s',a'))$ (3)
  
  which should be the Bellman equation for $q\_\pi(s,a)$, right?
  
  On the other hand, when connecting $q\_\pi(s,a)$ with $v\_\pi(s')$, [in this answer](https://ai.stackexchange.com/questions/21684/why-state-action-value-function-as-an-expected-value-of-the-return-and-state-val/21687#21687), I believe this is done
  
  $q\_\pi(s,a) = \sum\_{a\in A} \pi(a |s) \sum\_{s' \in S}\sum\_{r \in R} p(s',r|s,a)(r + \gamma E\_{\pi}[G\_{t+1} | S\_{t+1} = s'])$ (4)
  
  $q\_\pi(s,a) = \sum\_{a\in A} \pi(a |s) \sum\_{s' \in S}\sum\_{r \in R} p(s',r|s,a)(r + \gamma v\_\pi(s'))$ (5)
  
  Is the difference between using the expectation $E\_{\pi}[G\_{t+1} | S\_{t+1} = s', A\_{t+1} = a']$ in (3) and the expectation $E\_{\pi}[G\_{t+1} | S\_{t+1} = s']$ in $(4)$ simply the difference in how we choose to express the expected return $G\_{t+1}$ at $s'$ in the definition of $q\_\pi(s,a)$?
  
  In $3$, we express the total return at $s'$ using the action value function
  
  [![enter image description here](https://i.sstatic.net/D8jyk.png)](https://i.sstatic.net/D8jyk.png)
  
  leading to the recursion and the Bellman equation, and in $4$, the total return is expressed at $s'$ using the value function
  
  [![enter image description here](https://i.sstatic.net/rlxHz.png)](https://i.sstatic.net/rlxHz.png)
  
  leading to $q\_\pi(s,a) = q\_\pi(s,a,v\_\pi(s'))$?
  
  **Tags:** reinforcement-learning, value-functions, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-17. Score: 5 | Accepted answer
  
  Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $a$ in state $s$. Now, in your equation (2) you have conditioned on taking action $a'$ in the inner expectation - this is not what happens in the state-action value function, you *do not* condition on knowing $A\_{t+1}$, it is chosen according to the policy $\pi$ as per the definition of a bellman equation.
  
  If you want to see a 'recursion' between state action value functions, note that
  
  $$v\_\pi(s) = \sum\_a \pi(a|s)q\_\pi(s,a)\;,$$
  
  Your equation (5) is incorrect -- you need to drop the outter sum over $a$ as we have conditioned on knowing $a$. I will drop the $\pi$ subscripts for ease on notation, and we can see a 'recursion' for state-action value functions as:
  
  $$q(s,a) = \sum\_{s',r}p(s',r|s,a)\left(r + \gamma \left[\sum\_{a'} \pi(a'|s')q(s',a')\right]\right)\;.$$
  
  
  ---
  
  