<p>As <span class="math-container">$q_{\pi}(s,a)$</span> is defined in S&amp;B RL book as follows:</p>
<blockquote>
<p>we define the value of taking action a in state s under a policy <span class="math-container">$\pi$</span>, denoted <span class="math-container">$q_{\pi}(s,a)$</span>, as the expected return starting from <span class="math-container">$s$</span>, taking the action <span class="math-container">$a$</span>, and thereafter following policy <span class="math-container">$\pi$</span></p>
</blockquote>
<p>Therefore similar to state value, your highlighted step here is trying to represent your action value's <span class="math-container">$\mathbb{E}_{\pi}[G_{t+1}|S_t=s,A_t=a]$</span> component by the state values of all possible entering states <span class="math-container">$s'$</span> after taking action <span class="math-container">$a$</span> following the same policy <span class="math-container">$\pi$</span> thereafter. As for the other component involving <span class="math-container">$R_{t+1}$</span> you've probably already known it's only determined by the environment under above conditions.</p>

CC BY-SA 4.0
2024-02-28T19:09:49.847Z
2024-02-28T19:15:19.147Z
2024-02-28T19:15:19.147Z
cinch
<p>I did not understand how third equality follows from the second equality. Could some expert explain?</p>
<p><a href="https://i.sstatic.net/hjRtk.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hjRtk.jpg" alt="enter image description here" /></a></p>

<reinforcement-learning><sutton-barto>
Unclear point in derivation of action-value function
DSPinfinity
# Unclear point in derivation of action-value function

(Asked by: username_0 on 2024-02-28)

I did not understand how third equality follows from the second equality. Could some expert explain?

[![enter image description here](https://i.sstatic.net/hjRtk.jpg)](https://i.sstatic.net/hjRtk.jpg)

**Tags:** reinforcement-learning, sutton-barto


---

## 1 answers

### Answer by username_1 on 2024-02-28. Score: 2 | Accepted answer

As $q\_{\pi}(s,a)$ is defined in S&B RL book as follows:

> we define the value of taking action a in state s under a policy $\pi$, denoted $q\_{\pi}(s,a)$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$

Therefore similar to state value, your highlighted step here is trying to represent your action value's $\mathbb{E}\_{\pi}[G\_{t+1}|S\_t=s,A\_t=a]$ component by the state values of all possible entering states $s'$ after taking action $a$ following the same policy $\pi$ thereafter. As for the other component involving $R\_{t+1}$ you've probably already known it's only determined by the environment under above conditions.


---

