Answers.Body: <blockquote>
  <p>Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?</p>
  </blockquote>
  <p>You should probably re-start with <span class="math-container">$\epsilon$</span> at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.</p>
  <p>Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.</p>
  <p>From your description, I am assuming that you are monitoring the average reward per <em>training</em> episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using <span class="math-container">$\epsilon$</span>-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.</p>
  <blockquote>
  <p>Is this behavior expected?</p>
  </blockquote>
  <p>I would expect it if you re-set <span class="math-container">$\epsilon$</span> to a high value.</p>
  <blockquote>
  <p>If not, how do I properly resume training of a deep q-learning network?</p>
  </blockquote>
  <p>I recommend that you look at the following things:</p>
  <ul>
  <li><p>Re-start with <span class="math-container">$\epsilon$</span> at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.</p>
  </li>
  <li><p>Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.</p>
  </li>
  <li><p>Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.</p>
  </li>
  </ul>
  <p>You could consider these in priority order. Note that just the first one may appear to &quot;fix&quot; your problem, but that is because you are not yet properly measuring your agent's performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T04:43:04.810Z
Answers.Text: Thank you for your suggestions and the detailed explanation. As you have suggested, re-start at previous epsilon  has helped solved the problem. Regarding the performance measurement, would it be correct to say that as epsilon becomes quite small, the reward obtained in training episode should be quite close to test result?
Answers.UserDisplayName: Tianxun Zhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T06:44:28.667Z
Answers.Text: @TianxunZhou Yes, the expected reward for $\epsilon$-greedy policy will become close to expected reward for greedy policy as $\epsilon$ becomes close to zero. It is not always clear how close to zero you need to get before that is true - that depends on the environment.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-12T15:28:43.343Z
Answers.LastActivityDate: 2021-04-12T15:28:43.343Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a certain number of episodes and save the resultant model. Later, I will load up the previously saved model and resume training.</p>
  <p>However, what I'm noticing is that when training resumes, the average rewards goes back to very low again, compared to what it achieved at the end of the previous training session. What I'm currently doing is to load up the previously saved model into the prediction and target models, and I keep all hyperparameters unchanged.</p>
  <ul>
  <li><strong>Is this behaviour expected?</strong></li>
  <li><strong>If not, how do I properly resume training of a deep q-learning
  network?</strong></li>
  <li><strong>Do I start off with the epsilon value at the end of the
  previous session, currently I reinitialize that as well?</strong></li>
  </ul>
  
Tags: <reinforcement-learning><deep-learning><training><q-learning>
Title: How to properly resume training of deep Q-learning network?
UserDisplayName: Tianxun Zhou
ThreadText: # How to properly resume training of deep Q-learning network?
  
  (Asked by: username_0 on 2021-04-12)
  
  I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a certain number of episodes and save the resultant model. Later, I will load up the previously saved model and resume training.
  
  However, what I'm noticing is that when training resumes, the average rewards goes back to very low again, compared to what it achieved at the end of the previous training session. What I'm currently doing is to load up the previously saved model into the prediction and target models, and I keep all hyperparameters unchanged.
  
  * **Is this behaviour expected?**
  * **If not, how do I properly resume training of a deep q-learning
    network?**
  * **Do I start off with the epsilon value at the end of the
    previous session, currently I reinitialize that as well?**
  
  **Tags:** reinforcement-learning, deep-learning, training, q-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-12. Score: 1 | Accepted answer
  
  > Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?
  
  You should probably re-start with $\epsilon$ at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.
  
  Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.
  
  From your description, I am assuming that you are monitoring the average reward per *training* episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using $\epsilon$-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.
  
  > Is this behavior expected?
  
  I would expect it if you re-set $\epsilon$ to a high value.
  
  > If not, how do I properly resume training of a deep q-learning network?
  
  I recommend that you look at the following things:
  
  * Re-start with $\epsilon$ at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.
  * Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.
  * Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.
  
  You could consider these in priority order. Note that just the first one may appear to "fix" your problem, but that is because you are not yet properly measuring your agent's performance.
  
  > **Comments:**
  > **username_0**: Thank you for your suggestions and the detailed explanation. As you have suggested, re-start at previous epsilon  has helped solved the problem. Regarding the performance measurement, would it be correct to say that as epsilon becomes quite small, the reward obtained in training episode should be quite close to test result?  
  > **username_1**: @username_0 Yes, the expected reward for $\epsilon$-greedy policy will become close to expected reward for greedy policy as $\epsilon$ becomes close to zero. It is not always clear how close to zero you need to get before that is true - that depends on the environment.  
  
  ---
  
  