Answers.Body: <p>The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description.</p>
  <p>The large arrow shows dependency of data collection that's core to the whole process.</p>
  <p>The environment responds to actions chosen by the policy.</p>
  <p>The policy is informed only by data collected from the environment.</p>
  <p>The smaller arrows show data flow between sub-components, and how Dyna-Q is constructed to learn and improve policy. From a 10,000 foot view though, the large arrow reminds you that the usual interdependence of agent and environment is in play, and you still have usual impact of that - exploration Vs exploitation tradeoff for example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-10T07:44:28.520Z
Answers.LastActivityDate: 2024-05-10T07:44:28.520Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sutton-Barto, page 163:</p>
  <p><a href="https://i.sstatic.net/3D9eONlD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3D9eONlD.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between
  the environment and the policy, affects policy and value functions in much the same way as does simulated experience generated by the model of the environment.</p>
  </blockquote>
  <p>The central arrow is unnecessary in my opinion and moreover, the explanation for it is not clear to me. Could someone clarify it?
  Why real experience passing back and forth between the environment and the policy? The direction from &quot;real experience&quot; to &quot;environment&quot; is not clear to me. Moreover, the direction from &quot;real experience&quot; to &quot;Polcy/value functions&quot; is already covered in direct RL update.</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: Unclear arrow in general Dyna architecture
UserDisplayName: DSPinfinity
ThreadText: # Unclear arrow in general Dyna architecture
  
  (Asked by: username_0 on 2024-05-09)
  
  Sutton-Barto, page 163:
  
  [![enter image description here](https://i.sstatic.net/3D9eONlD.png)](https://i.sstatic.net/3D9eONlD.png)
  
  > Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between
  > the environment and the policy, affects policy and value functions in much the same way as does simulated experience generated by the model of the environment.
  
  The central arrow is unnecessary in my opinion and moreover, the explanation for it is not clear to me. Could someone clarify it?
  Why real experience passing back and forth between the environment and the policy? The direction from "real experience" to "environment" is not clear to me. Moreover, the direction from "real experience" to "Polcy/value functions" is already covered in direct RL update.
  
  **Tags:** reinforcement-learning, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-10. Score: 2 | Accepted answer
  
  The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description.
  
  The large arrow shows dependency of data collection that's core to the whole process.
  
  The environment responds to actions chosen by the policy.
  
  The policy is informed only by data collected from the environment.
  
  The smaller arrows show data flow between sub-components, and how Dyna-Q is constructed to learn and improve policy. From a 10,000 foot view though, the large arrow reminds you that the usual interdependence of agent and environment is in play, and you still have usual impact of that - exploration Vs exploitation tradeoff for example.
  
  
  ---
  
  