Body: <p>Sutton-Barto, page 174.</p>
  <p><a href="https://i.sstatic.net/MJCbIApB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MJCbIApB.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces
  the error to zero upon its completion.</p>
  </blockquote>
  <blockquote>
  <p>The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
  the real effect. In a real problem, the values of the successor states would be estimates
  that are themselves updated. By causing estimates to be more accurate sooner, sample
  updates will have a second advantage in that the values backed up from the successor
  states will be more accurate. These results suggest that sample updates are likely to be
  superior to expected updates on problems with large stochastic branching factors and
  too many states to be solved exactly.</p>
  </blockquote>
  <p>Question: How can Figure 8.7 probably be an underestimate of
  the real effect? Figure 8.7 is based on the fact that the value of successor states is correct. But in reality instead of these correct values estimated values will be used in updates. Then, it seems to me that Figure 8.7 is probably an overestimation of
  the real effect.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-16T21:06:23.030Z
Comments.Text: What are you reading/interpreting as "the effect" here?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-17T03:21:07.757Z
Comments.Text: I am very confused what the authors try to mean. It is not clear what they mean by effect but what I understand is effect is used to mean what happens in reality.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-17T06:14:27.637Z
Comments.Text: In the question, you are asking about (and disagreeing with the author) about "the effect" being smaller or larger.  From the last line of this question - "fig 8.7 is probably an overestimate of the effect" - when you use the words "the effect" in that sentence, what value, quantity or quality are **you** referring to (regardless of what the author is maybe referring to).
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-17T15:24:55.640Z
Comments.Text: my understanding by the word "effect" was that  authors meant in reality  RMS values in figure 8.7 would be lower when compared to the use of exact values of the next states. I was expecting the opposite of my understanding.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-17T15:30:14.847Z
Comments.Text: OK, so you are wrong about what the authors are referring to, but then correct in your reasoning about what would change (for the thing you are thinking of) when going from simplified to real. Could you add your understanding here to the question please, then I can answer
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><sutton-barto>
Title: Confusing statement in Sutton-Barto on expected versus sample updates
UserDisplayName: DSPinfinity
ThreadText: # Confusing statement in Sutton-Barto on expected versus sample updates
  
  (Asked by: username_0 on 2024-05-16)
  
  Sutton-Barto, page 174.
  
  [![enter image description here](https://i.sstatic.net/MJCbIApB.png)](https://i.sstatic.net/MJCbIApB.png)
  
  > b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces
  > the error to zero upon its completion.
  
  > The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
  > the real effect. In a real problem, the values of the successor states would be estimates
  > that are themselves updated. By causing estimates to be more accurate sooner, sample
  > updates will have a second advantage in that the values backed up from the successor
  > states will be more accurate. These results suggest that sample updates are likely to be
  > superior to expected updates on problems with large stochastic branching factors and
  > too many states to be solved exactly.
  
  Question: How can Figure 8.7 probably be an underestimate of
  the real effect? Figure 8.7 is based on the fact that the value of successor states is correct. But in reality instead of these correct values estimated values will be used in updates. Then, it seems to me that Figure 8.7 is probably an overestimation of
  the real effect.
  
  **Tags:** reinforcement-learning, sutton-barto
  > **Comments:**
  > **username_1**: What are you reading/interpreting as "the effect" here?  
  > **username_0**: I am very confused what the authors try to mean. It is not clear what they mean by effect but what I understand is effect is used to mean what happens in reality.  
  > **username_1**: In the question, you are asking about (and disagreeing with the author) about "the effect" being smaller or larger.  From the last line of this question - "fig 8.7 is probably an overestimate of the effect" - when you use the words "the effect" in that sentence, what value, quantity or quality are **you** referring to (regardless of what the author is maybe referring to).  
  > **username_0**: my understanding by the word "effect" was that  authors meant in reality  RMS values in figure 8.7 would be lower when compared to the use of exact values of the next states. I was expecting the opposite of my understanding.  
  > **username_1**: OK, so you are wrong about what the authors are referring to, but then correct in your reasoning about what would change (for the thing you are thinking of) when going from simplified to real. Could you add your understanding here to the question please, then I can answer  
  
  ---
  
  ## 0 answers
  
  