Answers.Body: <blockquote>
    <p>How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques?</p>
  </blockquote>
  
  <p>Using a recurrent neural network is one way for an agent to build a model of hidden or unobserved state in order to improve its predictions when direct observations do not give enough information, but a <em>history</em> of observations might give better information. Another way is to learn a <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model" rel="noreferrer">Hidden Markov model</a>. Both of these approaches build an internal representation that is effectively considered part of the state when making a decision by the agent. They are a way to approach solving <a href="https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process" rel="noreferrer">POMDPs</a>. </p>
  
  <p>You can consider using individual frame images from Atari games as state as a POMDP, because each individual frame does not contain information about velocity. Velocity of objects in play is an important concept in many video games. By formulating the problem as a POMDP with individual image inputs, this challenges the agent to find some representation of velocity (or something similar conceptually) from a sequence of images. Technically a NN may also do this using fixed inputs of 4 frames at a time (as per the original DQN Atari paper), but in that case the designers have deliberately "solved" the partially observable part of the problem for the agent in advance, by selecting a better state representation from the start.</p>
  
  <p>Experience replay solves some different problems:</p>
  
  <ul>
  <li><p>Efficient use of experience, by learning repeatedly from observed transitions. This is important when the agent needs to use a low learning rate, as it does when the environment has stochastic elements or when the agent includes a complex non-linear function approximator like a neural network.</p></li>
  <li><p>De-correlating samples to avoid problems with function approximators that work best with i.i.d. data. If you didn't effectively shuffle the dataset, the correlations between each time step could cause significant issues with a feed-forward neural network.</p></li>
  </ul>
  
  <p>These two issues are important to learning stability for neural networks in DQN. Without experience replay, often Q-learning with neural networks will fail to converge at all. </p>
  
  <blockquote>
    <p>Can they work together?</p>
  </blockquote>
  
  <p>Sort of, but not quite directly, because LSTM requires input of multiple related time steps at once, as opposed to randomly sampled individual time steps. However, you could keep a history of longer trajectories, and sample sections from it for the history in order to train a LSTM. This would still achieve the goal of using experience efficiently. Depending on the LSTM architecture, you may need to sample quite long trajectories or even complete episodes in order to do this.</p>
  
  <p>From comments by Muppet, it seems that is even possible to sample more randomly with individual steps by saving LSTM state. For instance, there is a paper "<a href="https://arxiv.org/pdf/1803.03916.pdf" rel="noreferrer">Deep reinforcement learning for time series: playing idealized trading games</a>" where the authors get a working system doing this. I have no experience of this approach myself, and there are theoretical reasons why this may not work in all cases, but it is an option.</p>
  
  <blockquote>
    <p>If they can work together, does it mean that the state is no longer a single state but a set of contiguous states?</p>
  </blockquote>
  
  <p>Not really, the state at any time step is still a single state representation, is separate conceptually from an observation, and is separate conceptually from a <em>trajectory</em> or sequence of states used to train a RNN (other RL approaches such as TD(<span class="math-container">$\lambda$</span>) also require longer trajectories). Using an LSTM implies you have <em>hidden</em> state on each time step (compared to what you are able to observe), and that you hope the LSTM will discover a way to represent it. </p>
  
  <p>One way to think of this is that the state is the current observation, plus a summary of observation history. The original Atari DQN paper simply used the previous three observations hard-coded as this "summary", which appeared to capture enough information to make predicting value functions reliable. </p>
  
  <p>The LSTM approach is partly of interest, because it does not rely on human input to decide how to construct state from the observations, but discovers this by itself. One key goal of deep learning is designs and architectures that are much less dependent on human interpretations of the problem (typically these use <em>feature engineering</em> to assist in learning process). An agent that can work directly from raw observations has solved more of the problem by itself without injection of knowledge by the engineers that built it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T15:56:43.553Z
Answers.Text: Neil, why is it that an LSTM has to be trained sequentially? R2D2 is doing the same. I managed to train pong with an LSTM where I'd simply save the (S, A, R, S') transition to the buffer each time with the hidden_state at the beginning of the transition. Then in training I would always reset the LSTM with the hidden_states before training the batch. Why do you think the sequential training is necessary?
Answers.UserDisplayName: Muppet
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T16:03:19.353Z
Answers.Text: In fact, here is another paper doing it like I did, with memory saved along the transitions: https://arxiv.org/pdf/1803.03916.pdf
Answers.UserDisplayName: Muppet
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T16:14:40.197Z
Answers.Text: @Muppet: Seems like a neat trick, wasn't aware you could do that. Although as the hidden state representation is supposed to be learnable in a RNN, then at least in theory it should not be reset back to what it experienced before as the current weights would lead it to some other hidden state if run without this reset and given the same experience. So there may be some compromises to performance doing that. If an experiment gets it to work well though, then it doesn't matter if this bends the theory somewhat
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-27T09:28:06.883Z
Answers.LastActivityDate: 2019-10-22T16:23:01.393Z
Answers.LastEditDate: 2019-10-22T16:23:01.393Z
Answers.UserDisplayName: Neil Slater
Body: <p>In the paper <a href="https://arxiv.org/pdf/1507.06527.pdf" rel="noreferrer">Deep Recurrent Q-Learning for Partially Observable MDPs</a>, the author processed the Atari game frames with an LSTM layer at the end. My questions are: </p>
  
  <ul>
  <li><p>How does this method differ from the <a href="https://datascience.stackexchange.com/questions/20535/understanding-experience-replay-in-reinforcement-learning">experience replay</a>, as they both use past information in the training? </p></li>
  <li><p>What's the typical application of both techniques? </p></li>
  <li><p>Can they work together?</p></li>
  <li><p>If they can work together, does it mean that the <em>state</em> is no longer a single state but a set of contiguous states?</p></li>
  </ul>
  
Tags: <reinforcement-learning><long-short-term-memory><deep-rl><comparison><experience-replay>
Title: How does LSTM in deep reinforcement learning differ from experience replay?
UserDisplayName: Kevin. Fang
ThreadText: # How does LSTM in deep reinforcement learning differ from experience replay?
  
  (Asked by: username_0 on 2018-08-27)
  
  In the paper [Deep Recurrent Q-Learning for Partially Observable MDPs](https://arxiv.org/pdf/1507.06527.pdf), the author processed the Atari game frames with an LSTM layer at the end. My questions are:
  
  * How does this method differ from the [experience replay](https://datascience.stackexchange.com/questions/20535/understanding-experience-replay-in-reinforcement-learning), as they both use past information in the training?
  * What's the typical application of both techniques?
  * Can they work together?
  * If they can work together, does it mean that the *state* is no longer a single state but a set of contiguous states?
  
  **Tags:** reinforcement-learning, long-short-term-memory, deep-rl, comparison, experience-replay
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-27. Score: 18 | Accepted answer
  
  > How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques?
  
  Using a recurrent neural network is one way for an agent to build a model of hidden or unobserved state in order to improve its predictions when direct observations do not give enough information, but a *history* of observations might give better information. Another way is to learn a [Hidden Markov model](https://en.wikipedia.org/wiki/Hidden_Markov_model). Both of these approaches build an internal representation that is effectively considered part of the state when making a decision by the agent. They are a way to approach solving [POMDPs](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process).
  
  You can consider using individual frame images from Atari games as state as a POMDP, because each individual frame does not contain information about velocity. Velocity of objects in play is an important concept in many video games. By formulating the problem as a POMDP with individual image inputs, this challenges the agent to find some representation of velocity (or something similar conceptually) from a sequence of images. Technically a NN may also do this using fixed inputs of 4 frames at a time (as per the original DQN Atari paper), but in that case the designers have deliberately "solved" the partially observable part of the problem for the agent in advance, by selecting a better state representation from the start.
  
  Experience replay solves some different problems:
  
  * Efficient use of experience, by learning repeatedly from observed transitions. This is important when the agent needs to use a low learning rate, as it does when the environment has stochastic elements or when the agent includes a complex non-linear function approximator like a neural network.
  * De-correlating samples to avoid problems with function approximators that work best with i.i.d. data. If you didn't effectively shuffle the dataset, the correlations between each time step could cause significant issues with a feed-forward neural network.
  
  These two issues are important to learning stability for neural networks in DQN. Without experience replay, often Q-learning with neural networks will fail to converge at all.
  
  > Can they work together?
  
  Sort of, but not quite directly, because LSTM requires input of multiple related time steps at once, as opposed to randomly sampled individual time steps. However, you could keep a history of longer trajectories, and sample sections from it for the history in order to train a LSTM. This would still achieve the goal of using experience efficiently. Depending on the LSTM architecture, you may need to sample quite long trajectories or even complete episodes in order to do this.
  
  From comments by username_2, it seems that is even possible to sample more randomly with individual steps by saving LSTM state. For instance, there is a paper "[Deep reinforcement learning for time series: playing idealized trading games](https://arxiv.org/pdf/1803.03916.pdf)" where the authors get a working system doing this. I have no experience of this approach myself, and there are theoretical reasons why this may not work in all cases, but it is an option.
  
  > If they can work together, does it mean that the state is no longer a single state but a set of contiguous states?
  
  Not really, the state at any time step is still a single state representation, is separate conceptually from an observation, and is separate conceptually from a *trajectory* or sequence of states used to train a RNN (other RL approaches such as TD($\lambda$) also require longer trajectories). Using an LSTM implies you have *hidden* state on each time step (compared to what you are able to observe), and that you hope the LSTM will discover a way to represent it.
  
  One way to think of this is that the state is the current observation, plus a summary of observation history. The original Atari DQN paper simply used the previous three observations hard-coded as this "summary", which appeared to capture enough information to make predicting value functions reliable.
  
  The LSTM approach is partly of interest, because it does not rely on human input to decide how to construct state from the observations, but discovers this by itself. One key goal of deep learning is designs and architectures that are much less dependent on human interpretations of the problem (typically these use *feature engineering* to assist in learning process). An agent that can work directly from raw observations has solved more of the problem by itself without injection of knowledge by the engineers that built it.
  
  > **Comments:**
  > **username_2**: username_1, why is it that an LSTM has to be trained sequentially? R2D2 is doing the same. I managed to train pong with an LSTM where I'd simply save the (S, A, R, S') transition to the buffer each time with the hidden_state at the beginning of the transition. Then in training I would always reset the LSTM with the hidden_states before training the batch. Why do you think the sequential training is necessary?  
  > **username_2**: In fact, here is another paper doing it like I did, with memory saved along the transitions: https://arxiv.org/pdf/1803.03916.pdf  
  > **username_1**: @username_2: Seems like a neat trick, wasn't aware you could do that. Although as the hidden state representation is supposed to be learnable in a RNN, then at least in theory it should not be reset back to what it experienced before as the current weights would lead it to some other hidden state if run without this reset and given the same experience. So there may be some compromises to performance doing that. If an experiment gets it to work well though, then it doesn't matter if this bends the theory somewhat  
  
  ---
  
  