Answers.Body: <p>As <span class="math-container">$q_{\pi}(s,a)$</span> is defined in S&amp;B RL book as follows:</p>
  <blockquote>
  <p>we define the value of taking action a in state s under a policy <span class="math-container">$\pi$</span>, denoted <span class="math-container">$q_{\pi}(s,a)$</span>, as the expected return starting from <span class="math-container">$s$</span>, taking the action <span class="math-container">$a$</span>, and thereafter following policy <span class="math-container">$\pi$</span></p>
  </blockquote>
  <p>Therefore similar to state value, your highlighted step here is trying to represent your action value's <span class="math-container">$\mathbb{E}_{\pi}[G_{t+1}|S_t=s,A_t=a]$</span> component by the state values of all possible entering states <span class="math-container">$s'$</span> after taking action <span class="math-container">$a$</span> following the same policy <span class="math-container">$\pi$</span> thereafter. As for the other component involving <span class="math-container">$R_{t+1}$</span> you've probably already known it's only determined by the environment under above conditions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-28T19:09:49.847Z
Answers.LastActivityDate: 2024-02-28T19:15:19.147Z
Answers.LastEditDate: 2024-02-28T19:15:19.147Z
Answers.UserDisplayName: cinch
Body: <p>I did not understand how third equality follows from the second equality. Could some expert explain?</p>
  <p><a href="https://i.sstatic.net/hjRtk.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hjRtk.jpg" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: Unclear point in derivation of action-value function
UserDisplayName: DSPinfinity
ThreadText: # Unclear point in derivation of action-value function
  
  (Asked by: username_0 on 2024-02-28)
  
  I did not understand how third equality follows from the second equality. Could some expert explain?
  
  [![enter image description here](https://i.sstatic.net/hjRtk.jpg)](https://i.sstatic.net/hjRtk.jpg)
  
  **Tags:** reinforcement-learning, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-28. Score: 2 | Accepted answer
  
  As $q\_{\pi}(s,a)$ is defined in S&B RL book as follows:
  
  > we define the value of taking action a in state s under a policy $\pi$, denoted $q\_{\pi}(s,a)$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$
  
  Therefore similar to state value, your highlighted step here is trying to represent your action value's $\mathbb{E}\_{\pi}[G\_{t+1}|S\_t=s,A\_t=a]$ component by the state values of all possible entering states $s'$ after taking action $a$ following the same policy $\pi$ thereafter. As for the other component involving $R\_{t+1}$ you've probably already known it's only determined by the environment under above conditions.
  
  
  ---
  
  