Answers.Body: <p>The hope in (f) is that the <span class="math-container">${max}_aQ(S^\prime,a)$</span> has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend the steps to experience it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-10T01:58:56.667Z
Answers.LastActivityDate: 2024-05-10T01:58:56.667Z
Answers.UserDisplayName: foreverska
Body: <p>Sutton-Barto, page 164:</p>
  <p><a href="https://i.sstatic.net/A1WzDa8J.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/A1WzDa8J.png" alt="enter image description here" /></a></p>
  <p>In the (f) loop, <span class="math-container">$S,A, S', R$</span> are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince).
  This experience is used in direct RL part in (d). Why is the same experience used
  again in the (f) part? Q-update in (d) uses the same info as Q-update in (f). I am totally confused.</p>
  <p>It seems to me that part (f) is a n-times re-application in parts before it.</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: Confusing point point in Dyna-Q
UserDisplayName: DSPinfinity
ThreadText: # Confusing point point in Dyna-Q
  
  (Asked by: username_0 on 2024-05-09)
  
  Sutton-Barto, page 164:
  
  [![enter image description here](https://i.sstatic.net/A1WzDa8J.png)](https://i.sstatic.net/A1WzDa8J.png)
  
  In the (f) loop, $S,A, S', R$ are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince).
  This experience is used in direct RL part in (d). Why is the same experience used
  again in the (f) part? Q-update in (d) uses the same info as Q-update in (f). I am totally confused.
  
  It seems to me that part (f) is a n-times re-application in parts before it.
  
  **Tags:** reinforcement-learning, sutton-barto
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-10. Score: 3 | Accepted answer
  
  The hope in (f) is that the ${max}\_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend the steps to experience it.
  
  
  ---
  
  