Answers.Body: <p>In general, <span class="math-container">$\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$</span>. <span class="math-container">$v_\pi(s)$</span> is defined as <span class="math-container">$\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$</span>, so you should be able to see why the two are not equal when the LHS is an expectation of the <span class="math-container">$n$</span>th step return. They would only be equal as <span class="math-container">$n \rightarrow \infty$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T14:54:36.707Z
Answers.LastActivityDate: 2020-09-15T14:54:36.707Z
Answers.UserDisplayName: David
Body: <p>Equation 7.3 of <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">Sutton Barto</a> book:
  <span class="math-container">$$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$</span>
  <span class="math-container">$$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2} + .....+\gamma^{n-1} R_{t+n} + \gamma^nV_{t+n-1}(S_{t+n})$$</span>
  Here <span class="math-container">$V_{t+n-1}(S_{t+n})$</span> is the estimate of <span class="math-container">$V_\pi(S_{t+n})$</span></p>
  <p>But the Left Hand Side of the above equation should be zeros as, for any state s, <span class="math-container">$G_{t:t+n}$</span> is an unbiased estimate of <span class="math-container">$v_\pi(s)$</span> hence <span class="math-container">$\mathbb{E}_\pi[G_{t:t+n}|S_t = s] = v_\pi(s)$</span>.</p>
  
Tags: <reinforcement-learning><value-functions><sutton-barto><expectation><return>
Title: What is wrong with equation 7.3 in Sutton & Barto's book?
UserDisplayName: ZERO NULLS
ThreadText: # What is wrong with equation 7.3 in Sutton & Barto's book?
  
  (Asked by: username_0 on 2020-09-15)
  
  Equation 7.3 of [Sutton Barto](http://incompleteideas.net/book/bookdraft2017nov5.pdf) book:
  $$\text{Equation: } max\_s|\mathbb{E}\_\pi[G\_{t:t+n}|S\_t = s] - v\_\pi| \le \gamma^nmax\_s|V\_{t+n-1}(s) - v\_\pi(s)| $$
  $$\text{where }G\_{t:t+n} = R\_{t+1} + \gamma R\_{t+2} + .....+\gamma^{n-1} R\_{t+n} + \gamma^nV\_{t+n-1}(S\_{t+n})$$
  Here $V\_{t+n-1}(S\_{t+n})$ is the estimate of $V\_\pi(S\_{t+n})$
  
  But the Left Hand Side of the above equation should be zeros as, for any state s, $G\_{t:t+n}$ is an unbiased estimate of $v\_\pi(s)$ hence $\mathbb{E}\_\pi[G\_{t:t+n}|S\_t = s] = v\_\pi(s)$.
  
  **Tags:** reinforcement-learning, value-functions, sutton-barto, expectation, return
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-15. Score: 3 | Accepted answer
  
  In general, $\mathbb{E}\_\pi[G\_{t:t+n}|S\_t = s] \neq v\_\pi(s)$. $v\_\pi(s)$ is defined as $\mathbb{E}\_\pi[\sum\_{k=0}^{\infty} \gamma^k R\_{t+k+1} | S\_t = s]$, so you should be able to see why the two are not equal when the LHS is an expectation of the $n$th step return. They would only be equal as $n \rightarrow \infty$.
  
  
  ---
  
  