Answers.Body: <p>Unlike algorithms presented in other chapters of Sutton and Barto, Dyna is a <em>planning</em> algorithm. That means that it makes decisions online, in a real environment, that attempt to be as optimal as possible given some constraints such as current knowledge and time available to compute between time steps. This differs from learning-only online algorithms which typically take a small step towards optimality on each piece of new experience as it happens.</p>
  
  <p>A planning algorithm can only do its job well if it is allowed to "look ahead" at the consequences of its behaviour whilst learning online. In fact, this is the <em>definition</em> of planning - to choose an action based on reasoning about consequences of that action. </p>
  
  <p>For an algorithm to look ahead before taking an action, it needs a model of how the environment will respond to that action. That model does not need to be coded up directly - e.g. you don't necessarily need to write a physics engine to predict the real world (although a basic one might be a good prior or pre-training step). Instead it can be a <em>learned</em> model, and typically in e.g. Dyna-Q, that is what you use.</p>
  
  <p>There is a strong relation between Dyna-Q, and regular Q-learning with experience replay. In the most basic forms, they are essentially the same algorithm with a different framing. However, you can take the planning ideas further e.g. focus improvements around the currently experienced state and paths to a goal state in Dyna-Q, perhaps making it closer to MCTS conceptually.</p>
  
  <blockquote>
    <p>Wouldn't it be more helpful to use real env instead of fake one?</p>
  </blockquote>
  
  <p>Most real environments do not let you take actions, see the consequences and then rewind in order to re-try. Essentially that is what planning algorithms are making up for - they try to predict consequences. This is important when mistakes made during training have real consequences, for example for a physical robot navigating an environment where there might be a possibility of a fall or collision that damaged something. Whilst online learning algorithms such as SARSA will also help with this in different ways (in SARSA by changing policy to allow for exploratory moves), typically Q-learning will be weaker than Dyna-Q when it comes to learning quickly from mistakes. With the usual caveat: Much still depends on the specific problem and choices of hyperparameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T08:42:58.540Z
Answers.Text: You said, "That means that it makes decisions online, in a real environment, that attempt to be optimal". Question. Do other algorithms, such as Monte Carlo, SARSA, Q-learning, etc, make decisions online as well?? What is the meaning of 'decision' you are talking about? Action? if so, all of those algorithms make decision online(every time it faced with a new state), don't they? (based on Q-value & e-greedy etc)
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T08:44:39.237Z
Answers.Text: You said "A 'planning algorithm' can only do its job well if it is allowed to 'look ahead' at the consequences of its behaviour. For that it needs a 'model'". Question. You said 'look ahead', but according to the Dyna algorithm, it makes (or improve) a `model` only with the states and actions it already took before.. Why is it considered as 'look ahead'?
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T08:45:33.620Z
Answers.Text: You said,  "There is a strong relation between "Dyna-Q" and 'regular Q-learning with experience replay'". Question. Now that after I saw your explanation, It seems like they are really look same.. I think that  'experience replay' completely corresponds to the `model` of `Dyna-Q`. What's the difference?
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T08:52:04.730Z
Answers.Text: You said, "This is important when mistakes made during training have real consequences, for example ~". Question. But in this real environment case, wouldn't the return of that (state, action) be getting updated to lower value everytime the robot experience that situation? Then as time goes, the robot would evade those action cuz it decide the action based on the Value? Also, even if I use a model, If I made a mistake in a real env, does it also affect the model to choose that action again?
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T08:59:29.183Z
Answers.Text: @user3595632: That's far too many questions. I cannot answer all those, I would be writing an essay. But essentially yes you have a point in your questions. If you read the chapter again, you will see that S&B call out the strong similarities between planning and learning. So, for example in your last question, you wonder what the difference is about mistakes. The answer is that a planning algorithm will often be *better* at avoiding mistakes after a single incident than a pure learning algorithm which may repeat them in order to learn enough.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T09:20:48.360Z
Answers.Text: @user3595632: I have updated my answer to try and address some of your questions/concerns.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T15:14:10.420Z
Answers.Text: Thanks for the really great explanation. Since  I cannot 100% understand it, I need to read ch08 all over again!
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-31T14:15:02.470Z
Answers.Text: @FauChristian Your slide chapter is different with that of book (2nd edition)... We talked about Planning and Learning !!
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-31T20:04:26.040Z
Answers.Text: @FauChristian I think OP's question and my answer are referring to 2nd edition of the book, and your course _may_ refer to 1st edition chapter numbers. You can find a freely-available PDF of 2nd edition here: http://incompleteideas.net/book/the-book-2nd.html
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-01T13:37:15.510Z
Answers.Text: @DouglasDaseeco: Yes a model of the environment has multiple uses/purposes. But OP's question is specifically about why it is *required* in Dyna. And it is only required to allow planning. The rest of the algorithm is model-free.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T06:55:15.647Z
Answers.LastActivityDate: 2018-05-29T09:28:34.047Z
Answers.LastEditDate: 2018-05-29T09:28:34.047Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Sutton's Dyna has been shown to be more effective for many problem spaces than learning systems that work without a model, yet it requires fewer processing cycles than certainty-equivalence methods.  It is advanced in that it, in parallel, builds a model and adjusts behavioral policy based on both incoming information and the model.  The goal was to integrate both identified capabilities of the human brain.</p>
  
  <p><em>Why do we need Model in Dyna?</em></p>
  
  <p>The capacity to model is an essential part of the Dyna architecture and may, over time, prove to be an essential component to achieve the greater effectiveness.  Many think so, myself included.  In other words, there may be no equivalently effective mechanism than building and maintaining a model for many problem sets.</p>
  
  <p><em>Why can't we just utilize a real environment itself?  Wouldn't it be more helpful to use real environment instead of fake one?</em></p>
  
  <p>The real environment cannot be placed in memory for many reasons.  Primarily, it would not fit.  Furthermore, not much of it can be acquired.  Only images of the environment can be acquired and placed in memory.  The most important characteristic of images, whether they be stock prices, temperature readings, or streamed video, is that they are grossly sparse and undifferentiated representations of the environment on which we are attempting to operate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-30T06:28:21.447Z
Answers.LastActivityDate: 2018-05-30T06:28:21.447Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>In chapter 8 of "Reinforcement Learning: An Introduction" by Sutton and Barto, it is stated that Dyna needs a model to simulate the environment. </p>
  
  <p>But why do we need a model? Why can't we just use the real environment itself? Wouldn't it be more helpful to use real environment instead of fake one?</p>
  
Tags: <reinforcement-learning><models>
Title: Why do we need a model of the environment in Dyna?
UserDisplayName: user3595632
ThreadText: # Why do we need a model of the environment in Dyna?
  
  (Asked by: username_0 on 2018-05-29)
  
  In chapter 8 of "Reinforcement Learning: An Introduction" by Sutton and Barto, it is stated that Dyna needs a model to simulate the environment.
  
  But why do we need a model? Why can't we just use the real environment itself? Wouldn't it be more helpful to use real environment instead of fake one?
  
  **Tags:** reinforcement-learning, models
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-05-29. Score: 3
  
  Unlike algorithms presented in other chapters of Sutton and Barto, Dyna is a *planning* algorithm. That means that it makes decisions online, in a real environment, that attempt to be as optimal as possible given some constraints such as current knowledge and time available to compute between time steps. This differs from learning-only online algorithms which typically take a small step towards optimality on each piece of new experience as it happens.
  
  A planning algorithm can only do its job well if it is allowed to "look ahead" at the consequences of its behaviour whilst learning online. In fact, this is the *definition* of planning - to choose an action based on reasoning about consequences of that action.
  
  For an algorithm to look ahead before taking an action, it needs a model of how the environment will respond to that action. That model does not need to be coded up directly - e.g. you don't necessarily need to write a physics engine to predict the real world (although a basic one might be a good prior or pre-training step). Instead it can be a *learned* model, and typically in e.g. Dyna-Q, that is what you use.
  
  There is a strong relation between Dyna-Q, and regular Q-learning with experience replay. In the most basic forms, they are essentially the same algorithm with a different framing. However, you can take the planning ideas further e.g. focus improvements around the currently experienced state and paths to a goal state in Dyna-Q, perhaps making it closer to MCTS conceptually.
  
  > Wouldn't it be more helpful to use real env instead of fake one?
  
  Most real environments do not let you take actions, see the consequences and then rewind in order to re-try. Essentially that is what planning algorithms are making up for - they try to predict consequences. This is important when mistakes made during training have real consequences, for example for a physical robot navigating an environment where there might be a possibility of a fall or collision that damaged something. Whilst online learning algorithms such as SARSA will also help with this in different ways (in SARSA by changing policy to allow for exploratory moves), typically Q-learning will be weaker than Dyna-Q when it comes to learning quickly from mistakes. With the usual caveat: Much still depends on the specific problem and choices of hyperparameters.
  
  > **Comments:**
  > **username_0**: You said, "That means that it makes decisions online, in a real environment, that attempt to be optimal". Question. Do other algorithms, such as Monte Carlo, SARSA, Q-learning, etc, make decisions online as well?? What is the meaning of 'decision' you are talking about? Action? if so, all of those algorithms make decision online(every time it faced with a new state), don't they? (based on Q-value & e-greedy etc)  
  > **username_0**: You said "A 'planning algorithm' can only do its job well if it is allowed to 'look ahead' at the consequences of its behaviour. For that it needs a 'model'". Question. You said 'look ahead', but according to the Dyna algorithm, it makes (or improve) a `model` only with the states and actions it already took before.. Why is it considered as 'look ahead'?  
  > **username_0**: You said,  "There is a strong relation between "Dyna-Q" and 'regular Q-learning with experience replay'". Question. Now that after I saw your explanation, It seems like they are really look same.. I think that  'experience replay' completely corresponds to the `model` of `Dyna-Q`. What's the difference?  
  > **username_0**: You said, "This is important when mistakes made during training have real consequences, for example ~". Question. But in this real environment case, wouldn't the return of that (state, action) be getting updated to lower value everytime the robot experience that situation? Then as time goes, the robot would evade those action cuz it decide the action based on the Value? Also, even if I use a model, If I made a mistake in a real env, does it also affect the model to choose that action again?  
  > **username_1**: @user3595632: That's far too many questions. I cannot answer all those, I would be writing an essay. But essentially yes you have a point in your questions. If you read the chapter again, you will see that S&B call out the strong similarities between planning and learning. So, for example in your last question, you wonder what the difference is about mistakes. The answer is that a planning algorithm will often be *better* at avoiding mistakes after a single incident than a pure learning algorithm which may repeat them in order to learn enough.  
  > **username_1**: @user3595632: I have updated my answer to try and address some of your questions/concerns.  
  > **username_0**: Thanks for the really great explanation. Since  I cannot 100% understand it, I need to read ch08 all over again!  
  > **username_0**: @FauChristian Your slide chapter is different with that of book (2nd edition)... We talked about Planning and Learning !!  
  > **username_1**: @FauChristian I think OP's question and my answer are referring to 2nd edition of the book, and your course _may_ refer to 1st edition chapter numbers. You can find a freely-available PDF of 2nd edition here: http://incompleteideas.net/book/the-book-2nd.html  
  > **username_1**: @DouglasDaseeco: Yes a model of the environment has multiple uses/purposes. But OP's question is specifically about why it is *required* in Dyna. And it is only required to allow planning. The rest of the algorithm is model-free.  
  
  ---
  
  ### Answer by username_2 on 2018-05-30. Score: 0
  
  Sutton's Dyna has been shown to be more effective for many problem spaces than learning systems that work without a model, yet it requires fewer processing cycles than certainty-equivalence methods. It is advanced in that it, in parallel, builds a model and adjusts behavioral policy based on both incoming information and the model. The goal was to integrate both identified capabilities of the human brain.
  
  *Why do we need Model in Dyna?*
  
  The capacity to model is an essential part of the Dyna architecture and may, over time, prove to be an essential component to achieve the greater effectiveness. Many think so, myself included. In other words, there may be no equivalently effective mechanism than building and maintaining a model for many problem sets.
  
  *Why can't we just utilize a real environment itself? Wouldn't it be more helpful to use real environment instead of fake one?*
  
  The real environment cannot be placed in memory for many reasons. Primarily, it would not fit. Furthermore, not much of it can be acquired. Only images of the environment can be acquired and placed in memory. The most important characteristic of images, whether they be stock prices, temperature readings, or streamed video, is that they are grossly sparse and undifferentiated representations of the environment on which we are attempting to operate.
  
  
  ---
  
  