Answers.Body: <p>Not quite. You are missing the reward at time step <span class="math-container">$t+1$</span>.</p>
  <p>The definition you are looking for is (leaving out the <span class="math-container">$\pi$</span> subscripts for ease of notation)</p>
  <p><span class="math-container">$$q(s,a) = \mathbb{E}[R_{t+1} + \gamma v(s') | S_t=s,A_t=a] = \sum_{r,s'}(r +\gamma v(s'))p(s',r|s,a)\;.$$</span></p>
  <p>Because <span class="math-container">$q(s,a)$</span> relates to expected returns at time <span class="math-container">$t$</span>, and returns are defined as <span class="math-container">$G_t = \sum_{b = 0}^\infty \gamma ^b R_{t+b+1}$</span>, thus <span class="math-container">$R_{t+1}$</span> is also a random variable at time <span class="math-container">$t$</span> that we need to take expectation with respect to, not just the state that we transition into.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T12:42:00.583Z
Answers.LastActivityDate: 2022-06-28T13:12:27.170Z
Answers.LastEditDate: 2022-06-28T13:12:27.170Z
Answers.UserDisplayName: David
Body: <p>The task (exercise 3.13 in the RL book by Sutton and Barto) is to express <span class="math-container">$q_\pi(s,a)$</span> as a function of <span class="math-container">$p(s',r|s,a)$</span> and <span class="math-container">$v_\pi(s)$</span>. </p>
  
  <p><span class="math-container">$q_\pi(s,a)$</span> is the action-value function, that states how good it is to be at some state <span class="math-container">$s$</span> in the Markov Decision Process (MDP), if at that state, we choose an action <span class="math-container">$a$</span>, and after that action, the policy <span class="math-container">$\pi(s,a)$</span> determines future actions.</p>
  
  <p>Say that we are at some state <span class="math-container">$s$</span>, and we choose an action <span class="math-container">$a$</span>. The probability of landing at some other state <span class="math-container">$s'$</span> is determined by <span class="math-container">$p(s',r|s,a)$</span>. Each new state <span class="math-container">$s'$</span> then has a state-value function that determines how good is it to be at <span class="math-container">$s'$</span> if all future actions are given by the policy <span class="math-container">$\pi(s',a)$</span>, therefore: </p>
  
  <p><span class="math-container">$$q_\pi(s,a) = \sum_{s' \in S} p(s',r|s,a) v_\pi(s')$$</span></p>
  
  <p>Is this correct?</p>
  
Tags: <reinforcement-learning><definitions><value-functions><sutton-barto>
Title: How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?
UserDisplayName: tmaric
ThreadText: # How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?
  
  (Asked by: username_0 on 2020-06-08)
  
  The task (exercise 3.13 in the RL book by Sutton and Barto) is to express $q\_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v\_\pi(s)$.
  
  $q\_\pi(s,a)$ is the action-value function, that states how good it is to be at some state $s$ in the Markov Decision Process (MDP), if at that state, we choose an action $a$, and after that action, the policy $\pi(s,a)$ determines future actions.
  
  Say that we are at some state $s$, and we choose an action $a$. The probability of landing at some other state $s'$ is determined by $p(s',r|s,a)$. Each new state $s'$ then has a state-value function that determines how good is it to be at $s'$ if all future actions are given by the policy $\pi(s',a)$, therefore:
  
  $$q\_\pi(s,a) = \sum\_{s' \in S} p(s',r|s,a) v\_\pi(s')$$
  
  Is this correct?
  
  **Tags:** reinforcement-learning, definitions, value-functions, sutton-barto
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-08. Score: 6 | Accepted answer
  
  Not quite. You are missing the reward at time step $t+1$.
  
  The definition you are looking for is (leaving out the $\pi$ subscripts for ease of notation)
  
  $$q(s,a) = \mathbb{E}[R\_{t+1} + \gamma v(s') | S\_t=s,A\_t=a] = \sum\_{r,s'}(r +\gamma v(s'))p(s',r|s,a)\;.$$
  
  Because $q(s,a)$ relates to expected returns at time $t$, and returns are defined as $G\_t = \sum\_{b = 0}^\infty \gamma ^b R\_{t+b+1}$, thus $R\_{t+1}$ is also a random variable at time $t$ that we need to take expectation with respect to, not just the state that we transition into.
  
  
  ---
  
  