Answers.Body: <p>Model-based RL is obviously the correct approach. Mainly because it lets you simulate the environment internally without having direct interaction.</p>
  <p>And all successful RL algorithms essentially are model-based because nobody has done real-time RL and been successful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T19:39:09.390Z
Answers.Text: Right, but I guess my question is about the idea of a model based RL in which agents keep information they see in "busy times" (a fight in the case of a shooter game) and then seek for idle times where they can better process that information, by adapting to it or even doing simulations over variations of it to improve their behavior.
Answers.UserDisplayName: Bernardo Subercaseaux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T21:47:00.987Z
Answers.Text: There isn't a concept of "idle time" for RL agents that I know of. Deep RL use the same power regardless of the current world.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T22:08:29.430Z
Answers.Text: Is Q-learning model-based? What was used to achieve super-human performance on the Atari games?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T22:26:19.943Z
Answers.Text: If the RL agent is running in a simulator to learn I would call the global approach to the problem effectively model based even if not explicit since the model has already been constructed.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T22:31:38.050Z
Answers.Text: @FourierFlux Well, the simulator is the game itself in the case of Atari and the DQN case. RL algorithms need an environment. So, according to your definition, every RL algorithm is model-based only because there's an environment and the distinction between model-based and model-free is meaningless. Please, have a look at [Neil's answer](https://ai.stackexchange.com/a/6733/2444) and [my answer](https://ai.stackexchange.com/a/8820/2444) to know the difference between model-based and model-free.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T22:44:36.003Z
Answers.Text: The Atari games tick speed has been dramatically increased to allow  many games to be played effectively meaning it is running in a simulator. I have yet to see any "model free" agent according to your definition learn anything in real time.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T23:49:42.587Z
Answers.Text: Model based RL is defined as an algorithm where an agent uses learned dynamics of the system to help with learning. Q-learning does not do this (not does DQN) and is strictly _model free_.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T19:34:41.377Z
Answers.LastActivityDate: 2020-08-26T22:07:58.053Z
Answers.LastEditDate: 2020-08-26T22:07:58.053Z
Answers.UserDisplayName: FourierFlux
Answers.Body: <p>Yes, the concept of <em>dreaming</em> or <em>imagining</em> has already been explored in reinforcement learning.</p>
  <p>For example, have a look at <a href="https://arxiv.org/pdf/1705.02670.pdf" rel="nofollow noreferrer">Metacontrol for Adaptive Imagination-Based Optimization</a> (2017) by Jessica B. Hamrick et al., which is a paper that I gave a talk/presentation on 1-2 years ago (though I don't remember well the details anymore).</p>
  <p>There is also a blog post about the topic <a href="https://deepmind.com/blog/article/agents-imagine-and-plan" rel="nofollow noreferrer">Agents that imagine and plan</a> (2017) by DeepMind, which discusses two more recent papers and also mentions Hamrick's paper.</p>
  <p>In 2018, another related and interesting paper was also presented at NIPS, i.e. <a href="https://worldmodels.github.io/" rel="nofollow noreferrer">World Models</a>, by Ha and Schmidhuber.</p>
  <p>If you search for &quot;imagination/dreaming in reinforcement learning&quot; on the web, you will find more papers and articles about this interesting topic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T22:17:21.330Z
Answers.LastActivityDate: 2020-08-26T23:19:44.113Z
Answers.LastEditDate: 2020-08-26T23:19:44.113Z
Answers.UserDisplayName: nbro
Body: <p>I was reading <a href="https://www.unil.ch/files/live/sites/ln/files/shared/Revonsuo_1.pdf" rel="nofollow noreferrer">this article</a> about the question &quot;Why do we dream?&quot; in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My question is <strong>whether this idea has been explored in the context of RL</strong>.</p>
  <p>For example, in a competition between AIs on a shooter game, one could design an agent that, besides the behavior it has learned in a &quot;normal&quot; training, seeks for time in which is out of danger, to then use its computation time in the game to produce simulations that would further optimize its behavior. As the agent still needs to be somewhat aware of its environment, it could alternate between processing the environment and this kind of simulation. Note that this &quot;in-game&quot; simulation has an advantage with respect to the &quot;pre-game&quot; simulations used for training; the agent in the game experiences the behavior of the other agents, which could not have been predicted beforehand, and then simulates on top of these experiences, e.g. by slightly modifying them.</p>
  <p><strong>For more experienced folks, does this idea make sense? has something similar been explored?</strong></p>
  <p>I have absolutely no experience in the field, so I apologize if this question is poorly worded, dumb or obvious. I would appreciate suggestions on how to improve it if this is the case.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T22:13:45.607Z
Comments.Text: It's not clear to me what the restriction really is that you're putting on the "imagination" strategy. Is the restriction that it needs to be performed online while the agent learns and interacts with the environment?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T23:55:29.067Z
Comments.Text: It seems to me dreaming, and sleep, wouldn't necessary for an algorithm, as the function we humans do while sleeping could likely be accomplished in parallel.  The closest thing I can think of, offhand, is "[goalless search](https://www.quantamagazine.org/computers-evolve-a-new-path-toward-human-intelligence-20191106/)", which is a form of algorithmic creativity.
Comments.UserDisplayName: DukeZhou
Tags: <reinforcement-learning><reference-request><model-based-methods><imagination>
Title: Have agents that "dream" been explored in Reinforcement Learning?
UserDisplayName: Bernardo Subercaseaux
ThreadText: # Have agents that "dream" been explored in Reinforcement Learning?
  
  (Asked by: username_0 on 2020-08-26)
  
  I was reading [this article](https://www.unil.ch/files/live/sites/ln/files/shared/Revonsuo_1.pdf) about the question "Why do we dream?" in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My question is **whether this idea has been explored in the context of RL**.
  
  For example, in a competition between AIs on a shooter game, one could design an agent that, besides the behavior it has learned in a "normal" training, seeks for time in which is out of danger, to then use its computation time in the game to produce simulations that would further optimize its behavior. As the agent still needs to be somewhat aware of its environment, it could alternate between processing the environment and this kind of simulation. Note that this "in-game" simulation has an advantage with respect to the "pre-game" simulations used for training; the agent in the game experiences the behavior of the other agents, which could not have been predicted beforehand, and then simulates on top of these experiences, e.g. by slightly modifying them.
  
  **For more experienced folks, does this idea make sense? has something similar been explored?**
  
  I have absolutely no experience in the field, so I apologize if this question is poorly worded, dumb or obvious. I would appreciate suggestions on how to improve it if this is the case.
  
  **Tags:** reinforcement-learning, reference-request, model-based-methods, imagination
  > **Comments:**
  > **username_1**: It's not clear to me what the restriction really is that you're putting on the "imagination" strategy. Is the restriction that it needs to be performed online while the agent learns and interacts with the environment?  
  > **username_2**: It seems to me dreaming, and sleep, wouldn't necessary for an algorithm, as the function we humans do while sleeping could likely be accomplished in parallel.  The closest thing I can think of, offhand, is "[goalless search](https://www.quantamagazine.org/computers-evolve-a-new-path-toward-human-intelligence-20191106/)", which is a form of algorithmic creativity.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-08-26. Score: 2 | Accepted answer
  
  Yes, the concept of *dreaming* or *imagining* has already been explored in reinforcement learning.
  
  For example, have a look at [Metacontrol for Adaptive Imagination-Based Optimization](https://arxiv.org/pdf/1705.02670.pdf) (2017) by Jessica B. Hamrick et al., which is a paper that I gave a talk/presentation on 1-2 years ago (though I don't remember well the details anymore).
  
  There is also a blog post about the topic [Agents that imagine and plan](https://deepmind.com/blog/article/agents-imagine-and-plan) (2017) by DeepMind, which discusses two more recent papers and also mentions Hamrick's paper.
  
  In 2018, another related and interesting paper was also presented at NIPS, i.e. [World Models](https://worldmodels.github.io/), by Ha and Schmidhuber.
  
  If you search for "imagination/dreaming in reinforcement learning" on the web, you will find more papers and articles about this interesting topic.
  
  
  ---
  
  ### Answer by username_3 on 2020-08-26. Score: 0
  
  Model-based RL is obviously the correct approach. Mainly because it lets you simulate the environment internally without having direct interaction.
  
  And all successful RL algorithms essentially are model-based because nobody has done real-time RL and been successful.
  
  > **Comments:**
  > **username_0**: Right, but I guess my question is about the idea of a model based RL in which agents keep information they see in "busy times" (a fight in the case of a shooter game) and then seek for idle times where they can better process that information, by adapting to it or even doing simulations over variations of it to improve their behavior.  
  > **username_3**: There isn't a concept of "idle time" for RL agents that I know of. Deep RL use the same power regardless of the current world.  
  > **username_1**: Is Q-learning model-based? What was used to achieve super-human performance on the Atari games?  
  > **username_3**: If the RL agent is running in a simulator to learn I would call the global approach to the problem effectively model based even if not explicit since the model has already been constructed.  
  > **username_1**: @FourierFlux Well, the simulator is the game itself in the case of Atari and the DQN case. RL algorithms need an environment. So, according to your definition, every RL algorithm is model-based only because there's an environment and the distinction between model-based and model-free is meaningless. Please, have a look at [Neil's answer](https://ai.stackexchange.com/a/6733/2444) and [my answer](https://ai.stackexchange.com/a/8820/2444) to know the difference between model-based and model-free.  
  > **username_3**: The Atari games tick speed has been dramatically increased to allow  many games to be played effectively meaning it is running in a simulator. I have yet to see any "model free" agent according to your definition learn anything in real time.  
  > **username_4**: Model based RL is defined as an algorithm where an agent uses learned dynamics of the system to help with learning. Q-learning does not do this (not does DQN) and is strictly _model free_.  
  
  ---
  
  