Answers.Body: <p>This research question seems to be analyzed in further details here (section 3) - <a href="https://openreview.net/pdf?id=r1lyTjAqYX" rel="nofollow noreferrer">https://openreview.net/pdf?id=r1lyTjAqYX</a>.</p>
  
  <p>Usually, a sequence is taken as a state to be fed into RNN to compute the final hidden state. One can then ask what initial state should the RNN be seeded with? This paper analyzes three methods with respect to the seed - </p>
  
  <ul>
  <li>zero initialization: When the RNN is initialized with the zero state </li>
  <li>burn-in: when the sequence is prepended by some preceding observations for RNN to learn a good initial state </li>
  <li>storing the initial hidden state: When the hidden state at the beginning of the sequence is stored</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-05T23:06:28.027Z
Answers.LastActivityDate: 2019-08-05T23:06:28.027Z
Answers.UserDisplayName: pg2455
Body: <p>I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like <a href="https://www.researchgate.net/publication/281670427_Recurrent_Reinforcement_Learning_A_Hybrid_Approach" rel="nofollow noreferrer">this</a> and <a href="https://papers.nips.cc/paper/1953-reinforcement-learning-with-long-short-term-memory.pdf" rel="nofollow noreferrer">this</a>. If I were to use a sequence of raw observations (or features) as an input to RNN for encoding the state of the system, I cannot change the weights of my network in the middle of the episode. Is that correct? Otherwise, the hidden state vectors will be different when the weights are changed. </p>
  
  <p>Does that mean that the use of RNN in RL has to store the entire episode before the weights can be changed?</p>
  
  <p>How does then one take into account the hidden states in RNN for RL? Are there any good tutorials on RNN-RL?</p>
  
Tags: <reinforcement-learning><recurrent-neural-networks><papers>
Title: How are the observations stored in the RNN that encodes the state?
UserDisplayName: pg2455
ThreadText: # How are the observations stored in the RNN that encodes the state?
  
  (Asked by: username_0 on 2019-08-03)
  
  I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like [this](https://www.researchgate.net/publication/281670427_Recurrent_Reinforcement_Learning_A_Hybrid_Approach) and [this](https://papers.nips.cc/paper/1953-reinforcement-learning-with-long-short-term-memory.pdf). If I were to use a sequence of raw observations (or features) as an input to RNN for encoding the state of the system, I cannot change the weights of my network in the middle of the episode. Is that correct? Otherwise, the hidden state vectors will be different when the weights are changed.
  
  Does that mean that the use of RNN in RL has to store the entire episode before the weights can be changed?
  
  How does then one take into account the hidden states in RNN for RL? Are there any good tutorials on RNN-RL?
  
  **Tags:** reinforcement-learning, recurrent-neural-networks, papers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-08-05. Score: 1
  
  This research question seems to be analyzed in further details here (section 3) - <https://openreview.net/pdf?id=r1lyTjAqYX>.
  
  Usually, a sequence is taken as a state to be fed into RNN to compute the final hidden state. One can then ask what initial state should the RNN be seeded with? This paper analyzes three methods with respect to the seed -
  
  * zero initialization: When the RNN is initialized with the zero state
  * burn-in: when the sequence is prepended by some preceding observations for RNN to learn a good initial state
  * storing the initial hidden state: When the hidden state at the beginning of the sequence is stored
  
  
  ---
  
  