Answers.Body: <p>The problem is the distribution of trajectories that your agent will observe, which changes during training</p>
  <p>LSTM at most might solve the &quot;i.i.d.&quot; assumption, however doing a parallelism to NLP, it's like if at the beginning of the training, you train your model on Wikipedia, then at some point you switch to Reddit, after a while to Twitter and so on</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-24T21:40:07.723Z
Answers.Text: Yeah but what I mean is that in some way, if the policy is an LSTM, the distribution of trajectories will be "even more" affected since some part of the input data is "directly" generated by the policy. I was wondering if this could be the cause of even greater instability.
Answers.UserDisplayName: SuperTardigrade
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-25T11:08:30.150Z
Answers.Text: @SuperTardigrade depends, if the LSTM sees that it only need the current observation (i.e. the Markov property holds for the environment), then it will learn to ignore the past
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-24T18:13:48.060Z
Answers.LastActivityDate: 2023-09-24T18:13:48.060Z
Answers.UserDisplayName: Alberto
Body: <p>We know the problem of &quot;distribution shift&quot; in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the collected data. This can cause instability during training.
  Since sequential models (like LSTM) generate their next input directly (at least partially), I was wondering if policies based on sequential models were even more sensitive to this phenomenon?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-03T13:13:17.277Z
Comments.Text: Can you explain what you mean by "policies based on sequential models"? By sequential models, do you mean based on RNNs?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-03T16:26:53.823Z
Comments.Text: Yes I mean policies like RNN, LSTM, GRU which would operate on sequence of observations. For instance:
  h_t, a_t = LSTM(x_t, h_{t-1}).
  Here the LSTM predicts the next action a_t but also predicts part of the next observation (h_t).
Comments.UserDisplayName: SuperTardigrade
Tags: <deep-rl><long-short-term-memory>
Title: Sequential models and distribution shift in RL
UserDisplayName: SuperTardigrade
ThreadText: # Sequential models and distribution shift in RL
  
  (Asked by: username_0 on 2023-09-24)
  
  We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the collected data. This can cause instability during training.
  Since sequential models (like LSTM) generate their next input directly (at least partially), I was wondering if policies based on sequential models were even more sensitive to this phenomenon?
  
  **Tags:** deep-rl, long-short-term-memory
  
  > **Comments:**
  > **username_1**: Can you explain what you mean by "policies based on sequential models"? By sequential models, do you mean based on RNNs?  
  > **username_0**: Yes I mean policies like RNN, LSTM, GRU which would operate on sequence of observations. For instance: h_t, a_t = LSTM(x_t, h_{t-1}). Here the LSTM predicts the next action a_t but also predicts part of the next observation (h_t).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-24. Score: 1
  
  The problem is the distribution of trajectories that your agent will observe, which changes during training
  
  LSTM at most might solve the "i.i.d." assumption, however doing a parallelism to NLP, it's like if at the beginning of the training, you train your model on Wikipedia, then at some point you switch to Reddit, after a while to Twitter and so on
  
  > **Comments:**
  > **username_0**: Yeah but what I mean is that in some way, if the policy is an LSTM, the distribution of trajectories will be "even more" affected since some part of the input data is "directly" generated by the policy. I was wondering if this could be the cause of even greater instability.  
  > **username_2**: @username_0 depends, if the LSTM sees that it only need the current observation (i.e. the Markov property holds for the environment), then it will learn to ignore the past  
  
  ---
  
  