Answers.Body: <blockquote>
  <p>If so, the results in Figure 8.7 for sample updates should be worse in real.</p>
  </blockquote>
  <p>Yes, the sample updates will perform worse than shown. The later paragraph is explaining that the results for expected updates may be even worse <em>relative</em> to sample updates. It is not claiming that sample updates are unaffected if you remove the simplifying assumption used in the comparison.</p>
  <p>The assertion on page 174 is that the disparity between the approaches may in practice persist for longer than the idealised situation (of already having correct values to backup/bootstrap from), because it may take longer to converge the expected values from the original estimates than the sampling approach would take. This is due to the same effect impacting each time step in a longer trajectory, whilst figure 8.7 shows the impact only for a single time step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T20:09:05.703Z
Answers.Text: @ Neil Slater, thanks for the answer. Sutton-Barto's explanation style is horrible in most places. Their English is also problematic in my opinion.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T17:45:21.110Z
Answers.LastActivityDate: 2022-03-23T17:45:21.110Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Sutton-Barto RL's book (page 174) it says:</p>
  <blockquote>
  <p>The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
  the real effect. In a real problem, the values of the successor states would be estimates
  that are themselves updated. By causing estimates to be more accurate sooner, sample
  updates will have a second advantage in that the values backed up from the successor
  states will be more accurate. These results suggest that sample updates are likely to be
  superior to expected updates on problems with large stochastic branching factors and
  too many states to be solved exactly.</p>
  </blockquote>
  <p>This is very confusing because before the above paragraph they say</p>
  <blockquote>
  <p>The values at the next states are assumed correct,...</p>
  </blockquote>
  <p>If so, the results in Figure 8.7 for sample updates should be worse in real.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-23T20:08:53.093Z
Comments.Text: If you can, please, put your **specific** question in the title.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning>
Title: Confusing statement in Sutton-Barto's RL book in Section 8.5 ( Expected vs. Sample Updates)
UserDisplayName: DSPinfinity
ThreadText: # Confusing statement in Sutton-Barto's RL book in Section 8.5 ( Expected vs. Sample Updates)
  
  (Asked by: username_0 on 2022-03-23)
  
  In Sutton-Barto RL's book (page 174) it says:
  
  > The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
  > the real effect. In a real problem, the values of the successor states would be estimates
  > that are themselves updated. By causing estimates to be more accurate sooner, sample
  > updates will have a second advantage in that the values backed up from the successor
  > states will be more accurate. These results suggest that sample updates are likely to be
  > superior to expected updates on problems with large stochastic branching factors and
  > too many states to be solved exactly.
  
  This is very confusing because before the above paragraph they say
  
  > The values at the next states are assumed correct,...
  
  If so, the results in Figure 8.7 for sample updates should be worse in real.
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: If you can, please, put your **specific** question in the title.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-23. Score: 1 | Accepted answer
  
  > If so, the results in Figure 8.7 for sample updates should be worse in real.
  
  Yes, the sample updates will perform worse than shown. The later paragraph is explaining that the results for expected updates may be even worse *relative* to sample updates. It is not claiming that sample updates are unaffected if you remove the simplifying assumption used in the comparison.
  
  The assertion on page 174 is that the disparity between the approaches may in practice persist for longer than the idealised situation (of already having correct values to backup/bootstrap from), because it may take longer to converge the expected values from the original estimates than the sampling approach would take. This is due to the same effect impacting each time step in a longer trajectory, whilst figure 8.7 shows the impact only for a single time step.
  
  > **Comments:**
  > **username_0**: @ Neil Slater, thanks for the answer. Sutton-Barto's explanation style is horrible in most places. Their English is also problematic in my opinion.  
  
  ---
  
  