Answers.Body: <p>The figure is of a broad architecture &quot;Dyna&quot;.  Of which, Dyna-Q is one such variant.  So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it.  On a previous page it is stated &quot;The model-learning method is also table-based and assumes the environment is deterministic.&quot;  If it is indeed deterministic, a single sample of the environment is a model of the environment for that state/action.  &quot;Simulated&quot; in this case is a perfect simulation of a state the agent is not presently in.</p>
  <p>Dyna-Q applied to a non-deterministic environment would of course model the environment poorly and depending on the amount of non-determinism would not be suitable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-10T01:27:53.573Z
Answers.LastActivityDate: 2024-05-10T01:27:53.573Z
Answers.UserDisplayName: foreverska
Answers.Body: <p>The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state.</p>
  <p>A random guess is technically a model, but normally not a useful one.</p>
  <p>A reference list of previous results is a useful model when the same inputs can be reused. Dyna-Q deliberately targets the known parts of the model for &quot;background planning&quot;.</p>
  <p>As an aside, the process is almost identical to experience replay used in DQN method, where the &quot;model&quot; is renamed &quot;replay buffer&quot;. This is done for partially different motivation than Dyna-Q, but it does show a strong conceptual link between planning and learning processes in reinforcement learning. In fact it's often not clear whether you should call some use of RL learning or planning - they are labels describing intent perhaps, and not necessarily strictly separate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-10T07:33:44.650Z
Answers.LastActivityDate: 2024-05-10T07:33:44.650Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sutton-Barto, page 164:</p>
  <p><a href="https://i.sstatic.net/p2aj23fg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/p2aj23fg.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/fzVVTIQ6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fzVVTIQ6.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>In the pseudocode algorithm for
  Dyna-Q in the box below, Model(s, a) denotes the contents of the model (predicted
  next state and reward) for state–action pair (s, a). Direct reinforcement learning, model learning,
  and planning are implemented by steps (d), (e), and (f), respectively. If (e) and
  (f) were omitted, the remaining algorithm would be one-step tabular Q-learning</p>
  </blockquote>
  <p>If we look at Dyna-Q algorithm in the second figure, we see that &quot;model learning&quot; part (part e) is simply recording the real transitions and hence this is definitely not learning a model. If you do not learn a model, then how can we use the terms &quot;model predictions&quot; and  &quot;simulated experience&quot; in the first figure? The simulated experience is the same as the real experience.</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions
UserDisplayName: DSPinfinity
ThreadText: # Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions
  
  (Asked by: username_0 on 2024-05-10)
  
  Sutton-Barto, page 164:
  
  [![enter image description here](https://i.sstatic.net/p2aj23fg.png)](https://i.sstatic.net/p2aj23fg.png)
  
  [![enter image description here](https://i.sstatic.net/fzVVTIQ6.png)](https://i.sstatic.net/fzVVTIQ6.png)
  
  > In the pseudocode algorithm for
  > Dyna-Q in the box below, Model(s, a) denotes the contents of the model (predicted
  > next state and reward) for state–action pair (s, a). Direct reinforcement learning, model learning,
  > and planning are implemented by steps (d), (e), and (f), respectively. If (e) and
  > (f) were omitted, the remaining algorithm would be one-step tabular Q-learning
  
  If we look at Dyna-Q algorithm in the second figure, we see that "model learning" part (part e) is simply recording the real transitions and hence this is definitely not learning a model. If you do not learn a model, then how can we use the terms "model predictions" and "simulated experience" in the first figure? The simulated experience is the same as the real experience.
  
  **Tags:** reinforcement-learning, sutton-barto
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-05-10. Score: 2 | Accepted answer
  
  The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it. On a previous page it is stated "The model-learning method is also table-based and assumes the environment is deterministic." If it is indeed deterministic, a single sample of the environment is a model of the environment for that state/action. "Simulated" in this case is a perfect simulation of a state the agent is not presently in.
  
  Dyna-Q applied to a non-deterministic environment would of course model the environment poorly and depending on the amount of non-determinism would not be suitable.
  
  
  ---
  
  ### Answer by username_2 on 2024-05-10. Score: 1
  
  The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state.
  
  A random guess is technically a model, but normally not a useful one.
  
  A reference list of previous results is a useful model when the same inputs can be reused. Dyna-Q deliberately targets the known parts of the model for "background planning".
  
  As an aside, the process is almost identical to experience replay used in DQN method, where the "model" is renamed "replay buffer". This is done for partially different motivation than Dyna-Q, but it does show a strong conceptual link between planning and learning processes in reinforcement learning. In fact it's often not clear whether you should call some use of RL learning or planning - they are labels describing intent perhaps, and not necessarily strictly separate.
  
  
  ---
  
  