Answers.Body: <blockquote>
    <p>The speaker argued that a dream is a random addition of memories, just as experience replay.</p>
  </blockquote>
  
  <p>The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. We don't even have strong consensus on why sleep is a necessary feature of animals, let alone what part dreaming plays in it. However, there are some widely-accepted theories, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3079906/" rel="noreferrer">with supporting evidence, that dreams are part of a learning and memorisation process</a>. Studies that manipulate sleep or dreaming have shown changes in the speed that skills are learned for example.</p>
  
  <p>Experience replay in reinforcement learning is a far more precise and well-understood affair, whereby individual time steps that occurred in the past are visited and re-assessed in light of current knowledge about long-term value, at random. If dreams were really like experience replay as it is practiced in RL today, then they would consist of a random jumble of tiny seemingly inconsequential events strung together, and all taken very exactly from the events of the past day. <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0185262" rel="noreferrer"><em>Sometimes</em> dreams do contain content like this</a>, but typically the content is far more varied.</p>
  
  <p>Taken with a large dose of artistic license, then yes, the speaker is referring to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4704085/" rel="noreferrer">real theories and conjectures about dreaming, that do have scientific support</a>. Although it is equally good to draw parallels between dreams and a higher-level <em>management</em> of the memory or experience replay data - which items to replay, and which to keep, depending on what is salient about the information. For instance, there is <a href="http://www.jneurosci.org/content/37/3/464" rel="noreferrer">good evidence that dreams help filter what is <em>forgotten</em></a>, and also evidence that events associated with strong emotional state are more likely to feature in dreams.</p>
  
  <p>It is important to separate the speaker's analogy, and any suggestion that a current reinforcement learning agent has a subjective experience. We are still a long way away from anything like that, and other similar use of a dreaming metaphor in machine learning - e.g. "Deep Dream" - is equally not an assertion that the devices are having an experience of any kind.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-10T06:20:16.787Z
Answers.LastActivityDate: 2018-09-10T09:24:38.737Z
Answers.LastEditDate: 2018-09-10T09:24:38.737Z
Answers.UserDisplayName: Neil Slater
Body: <p>Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no precaution, it can lead to major misunderstandings.</p>
  <p>I was recently attending a conference where the speaker described Experience Replay in RL as a way of making the net &quot;dream&quot;.
  I'm wondering how true this assertion is. The speaker argued that a dream is a random addition of memories, just as experience replay. However, I doubt the brain remembers its dream or either learns from it. What is your analysis?</p>
  
Tags: <reinforcement-learning><dqn><deep-rl><experience-replay>
Title: Is Experience Replay like dreaming?
UserDisplayName: 16Aghnar
ThreadText: # Is Experience Replay like dreaming?
  
  (Asked by: username_0 on 2018-09-09)
  
  Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no precaution, it can lead to major misunderstandings.
  
  I was recently attending a conference where the speaker described Experience Replay in RL as a way of making the net "dream".
  I'm wondering how true this assertion is. The speaker argued that a dream is a random addition of memories, just as experience replay. However, I doubt the brain remembers its dream or either learns from it. What is your analysis?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, experience-replay
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-10. Score: 8 | Accepted answer
  
  > The speaker argued that a dream is a random addition of memories, just as experience replay.
  
  The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. We don't even have strong consensus on why sleep is a necessary feature of animals, let alone what part dreaming plays in it. However, there are some widely-accepted theories, [with supporting evidence, that dreams are part of a learning and memorisation process](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3079906/). Studies that manipulate sleep or dreaming have shown changes in the speed that skills are learned for example.
  
  Experience replay in reinforcement learning is a far more precise and well-understood affair, whereby individual time steps that occurred in the past are visited and re-assessed in light of current knowledge about long-term value, at random. If dreams were really like experience replay as it is practiced in RL today, then they would consist of a random jumble of tiny seemingly inconsequential events strung together, and all taken very exactly from the events of the past day. [*Sometimes* dreams do contain content like this](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0185262), but typically the content is far more varied.
  
  Taken with a large dose of artistic license, then yes, the speaker is referring to [real theories and conjectures about dreaming, that do have scientific support](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4704085/). Although it is equally good to draw parallels between dreams and a higher-level *management* of the memory or experience replay data - which items to replay, and which to keep, depending on what is salient about the information. For instance, there is [good evidence that dreams help filter what is *forgotten*](http://www.jneurosci.org/content/37/3/464), and also evidence that events associated with strong emotional state are more likely to feature in dreams.
  
  It is important to separate the speaker's analogy, and any suggestion that a current reinforcement learning agent has a subjective experience. We are still a long way away from anything like that, and other similar use of a dreaming metaphor in machine learning - e.g. "Deep Dream" - is equally not an assertion that the devices are having an experience of any kind.
  
  
  ---
  
  