Answers.Body: <p>It is likely converging to a far worse local optimum from which it can't recover, so yes I would guess if all else is the same, that is where the issue would be. I would first try to adjust hyperparameters from a static seed for weight initialization.  ε and α, are likely good places to start. </p>
  
  <p>If that fails, adjust some architecture params like the number of units or layers in the network as well as the starting seed value. </p>
  
  <p>You seemingly have run into a core problem at the heart of ML research which is reproducibility, which is often predicated by initialization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T16:43:27.060Z
Answers.Text: Thanks for the suggestion. Let me recap if I got it right. 1- I do not care about reproducibility "per se" but I care in view of having more insights about the dynamics of learning (that I didnt saved), so that for the new runs with the different normalization method, I would have a comparison in terms of hours of training, evolution of q-values, loss etc. So, given the weights issue in order to reproduce the good experiment I would have to run a bunch of new simulations and, hopefully, get there again. But at this point shouldn't I rather start experimenting with the new normalization method?
Answers.UserDisplayName: FS93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T16:47:55.173Z
Answers.Text: 2- About the hyperparameters: should I keep the seed constant and then compare different values of alpha (I dont need epsilon as I can get rewards for every actions at each step), or should I keep alpha (and the other hyperparameters) the same as in the good run and just change different seeds till I get a new optimal result?
Answers.UserDisplayName: FS93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T16:53:02.523Z
Answers.Text: Yea, you can do it either way. I would intuitively just adjust hyperparams as if initialization is an issue, it likely means your model is also not optimal to solve the problem. However, you could just as easily play around with seed values to also reconverge.
Answers.UserDisplayName: hisairnessag3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T16:53:11.100Z
Answers.Text: 3- A bit off-topic: I normalized the database by dividing it into blocks of 96 steps and normalizing each block (independently) with its own Zscore. This is good for testing on another database but not viable for an online real application, where I get one element of the timeseries at the time, thus I have to change the normalization in a way that each element is normalized only according to its past data points. I am afraid that this might be detrimental to the performances as I would not be "leaking" any bit of information about the future. Do you think this can be a big issue?
Answers.UserDisplayName: FS93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T16:55:55.400Z
Answers.Text: For your 3rd point are you only doing this via experience replay or in the normal training loop?
Answers.UserDisplayName: hisairnessag3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T17:01:14.017Z
Answers.Text: I'm not sure what you mean. I divide the database into episodes and then I save an experience replay keeping track of the last 480 steps (inside each episode). Learning si done by updating the gradient each step from when the memory is full till the end of the episode (so 240 times per episode). For the training it samples a different window of 96 steps randomly from the memory of 480 each gradient update. So in the case of normalizing batches of 96, when it trains it can get the window at any point, not necessarily where the first element of the normalized block was. I hope it was clear.
Answers.UserDisplayName: FS93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T17:06:23.230Z
Answers.Text: Ah ok, so you aren't doing the 96 steps merge during environmental training. In that case there should be no issue. By that I mean, in TD learning for example, we bootstrap during an episode of training. However, it only seems like you are using that nuiance during replay(which is good).
Answers.UserDisplayName: hisairnessag3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T17:18:21.750Z
Answers.Text: I hope so, the merge is done before on the whole dataset and then I sample twice from the data, once for the episode and once for each gradient update. The agent should indeed get mixed enough data, thus not being able to understand when one block started and exploit this knowledge. Thanks a lot for your help, I will get back soon with, hopefully, good results!
Answers.UserDisplayName: FS93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T18:17:04.697Z
Answers.Text: Cool, go ahead and accept my answer if you deem it conclusive. Ill go ahead and answer anymore questions you have in the comments here when I see them
Answers.UserDisplayName: hisairnessag3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-02T16:32:12.813Z
Answers.LastActivityDate: 2019-04-02T16:32:12.813Z
Answers.UserDisplayName: hisairnessag3
Body: <p>I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for the first 480 steps without training, collecting a replay memory, and then update the parameters each step for the subsequent 240 steps using a window size (of 96 steps) randomly sampled from the replay memory (that always saves the last 480). </p>
  
  <p>My problem is that so far the agent learned the optimal policy just once, and it looks like this<a href="https://i.sstatic.net/CFyfg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CFyfg.png" alt="Optimal Q-values"></a>
  (on the test set, training is off) where, as you can see, the agent dynamically changes its evaluation of the state and acts greedily accordingly. All works fine and the performances are optimal, however, I have to slightly change the normalization of the database and rerun the training to get new parameters fitted to the new database. </p>
  
  <p>Trying to get to the same result has been proven impossible so far, (even keeping all settings the same!) because most of the time the agent learns to keep its q-values static such as <a href="https://i.sstatic.net/lNut3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lNut3.png" alt="that"></a>. 
  Note: this is an extract of the end of an episode and the beginnig of a new one, the noisy behaviour at the extreme is due to the model being trained for each step, in the middle the training is off and the agent acts greedily (as I explained above). The problem is that the learned parameters give rise to static q values that do not change while the state does and inevitably make the agent stuck in suboptimal strategy, as they never changes actions, even on longer sequences on the test set. The middle part of the second picture, where trainig is off, should look like the first picture, however, I am unable to get back to that optimal behaviour, even keeping all the parameters as they were. </p>
  
  <p>Any idea on what can be the cause of this anomalous behaviour? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-02T15:53:15.387Z
Comments.Text: Are you initializing the agent in a different way each time? I.e a different initial state
Comments.UserDisplayName: hisairnessag3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-02T16:11:23.567Z
Comments.Text: @hisairnessag3 No, in this case everything is kept the same. Hyperparameters, normalization of the database, optimizer (Nadam). The resulting graphs are obtained after a few hours of training (10-15 at least). I simply cannot reach the same optimal policy in the first place, even keeping all constant, and even less if I   change the normalization of the data.
Comments.UserDisplayName: FS93
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-02T16:12:23.010Z
Comments.Text: It seems your weight initialization is likely dynamic in some way then, if not an environment based issue
Comments.UserDisplayName: hisairnessag3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-02T16:16:44.157Z
Comments.Text: Oh, you meant the weight initialization. Yes, I actually initialize the LSTM layers with he_normal initialization (without keeping track of the seed unfortunately) and the final dense layer with normal distribution. Do you think that this stocasticity can be that detrimental in most cases? I thought that on the long run training would "flatten" any or most initial differences. If not, what do you suggest me to do?
Comments.UserDisplayName: FS93
Tags: <reinforcement-learning><q-learning><long-short-term-memory><dqn><convergence>
Title: DQN Q-values are static
UserDisplayName: FS93
ThreadText: # DQN Q-values are static
  
  (Asked by: username_0 on 2019-04-02)
  
  I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for the first 480 steps without training, collecting a replay memory, and then update the parameters each step for the subsequent 240 steps using a window size (of 96 steps) randomly sampled from the replay memory (that always saves the last 480).
  
  My problem is that so far the agent learned the optimal policy just once, and it looks like this[![Optimal Q-values](https://i.sstatic.net/CFyfg.png)](https://i.sstatic.net/CFyfg.png)
  (on the test set, training is off) where, as you can see, the agent dynamically changes its evaluation of the state and acts greedily accordingly. All works fine and the performances are optimal, however, I have to slightly change the normalization of the database and rerun the training to get new parameters fitted to the new database.
  
  Trying to get to the same result has been proven impossible so far, (even keeping all settings the same!) because most of the time the agent learns to keep its q-values static such as [![that](https://i.sstatic.net/lNut3.png)](https://i.sstatic.net/lNut3.png).
  Note: this is an extract of the end of an episode and the beginnig of a new one, the noisy behaviour at the extreme is due to the model being trained for each step, in the middle the training is off and the agent acts greedily (as I explained above). The problem is that the learned parameters give rise to static q values that do not change while the state does and inevitably make the agent stuck in suboptimal strategy, as they never changes actions, even on longer sequences on the test set. The middle part of the second picture, where trainig is off, should look like the first picture, however, I am unable to get back to that optimal behaviour, even keeping all the parameters as they were.
  
  Any idea on what can be the cause of this anomalous behaviour?
  
  **Tags:** reinforcement-learning, q-learning, long-short-term-memory, dqn, convergence
  > **Comments:**
  > **username_1**: Are you initializing the agent in a different way each time? I.e a different initial state  
  > **username_0**: @hisairnessag3 No, in this case everything is kept the same. Hyperparameters, normalization of the database, optimizer (Nadam). The resulting graphs are obtained after a few hours of training (10-15 at least). I simply cannot reach the same optimal policy in the first place, even keeping all constant, and even less if I   change the normalization of the data.  
  > **username_1**: It seems your weight initialization is likely dynamic in some way then, if not an environment based issue  
  > **username_0**: Oh, you meant the weight initialization. Yes, I actually initialize the LSTM layers with he_normal initialization (without keeping track of the seed unfortunately) and the final dense layer with normal distribution. Do you think that this stocasticity can be that detrimental in most cases? I thought that on the long run training would "flatten" any or most initial differences. If not, what do you suggest me to do?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-02. Score: 1 | Accepted answer
  
  It is likely converging to a far worse local optimum from which it can't recover, so yes I would guess if all else is the same, that is where the issue would be. I would first try to adjust hyperparameters from a static seed for weight initialization. ε and α, are likely good places to start.
  
  If that fails, adjust some architecture params like the number of units or layers in the network as well as the starting seed value.
  
  You seemingly have run into a core problem at the heart of ML research which is reproducibility, which is often predicated by initialization.
  
  > **Comments:**
  > **username_0**: Thanks for the suggestion. Let me recap if I got it right. 1- I do not care about reproducibility "per se" but I care in view of having more insights about the dynamics of learning (that I didnt saved), so that for the new runs with the different normalization method, I would have a comparison in terms of hours of training, evolution of q-values, loss etc. So, given the weights issue in order to reproduce the good experiment I would have to run a bunch of new simulations and, hopefully, get there again. But at this point shouldn't I rather start experimenting with the new normalization method?  
  > **username_0**: 2- About the hyperparameters: should I keep the seed constant and then compare different values of alpha (I dont need epsilon as I can get rewards for every actions at each step), or should I keep alpha (and the other hyperparameters) the same as in the good run and just change different seeds till I get a new optimal result?  
  > **username_1**: Yea, you can do it either way. I would intuitively just adjust hyperparams as if initialization is an issue, it likely means your model is also not optimal to solve the problem. However, you could just as easily play around with seed values to also reconverge.  
  > **username_0**: 3- A bit off-topic: I normalized the database by dividing it into blocks of 96 steps and normalizing each block (independently) with its own Zscore. This is good for testing on another database but not viable for an online real application, where I get one element of the timeseries at the time, thus I have to change the normalization in a way that each element is normalized only according to its past data points. I am afraid that this might be detrimental to the performances as I would not be "leaking" any bit of information about the future. Do you think this can be a big issue?  
  > **username_1**: For your 3rd point are you only doing this via experience replay or in the normal training loop?  
  > **username_0**: I'm not sure what you mean. I divide the database into episodes and then I save an experience replay keeping track of the last 480 steps (inside each episode). Learning si done by updating the gradient each step from when the memory is full till the end of the episode (so 240 times per episode). For the training it samples a different window of 96 steps randomly from the memory of 480 each gradient update. So in the case of normalizing batches of 96, when it trains it can get the window at any point, not necessarily where the first element of the normalized block was. I hope it was clear.  
  > **username_1**: Ah ok, so you aren't doing the 96 steps merge during environmental training. In that case there should be no issue. By that I mean, in TD learning for example, we bootstrap during an episode of training. However, it only seems like you are using that nuiance during replay(which is good).  
  > **username_0**: I hope so, the merge is done before on the whole dataset and then I sample twice from the data, once for the episode and once for each gradient update. The agent should indeed get mixed enough data, thus not being able to understand when one block started and exploit this knowledge. Thanks a lot for your help, I will get back soon with, hopefully, good results!  
  > **username_1**: Cool, go ahead and accept my answer if you deem it conclusive. Ill go ahead and answer anymore questions you have in the comments here when I see them  
  
  ---
  
  