Answers.Body: <p>Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning.</p>
  <p>Looking at it in detail I can say:</p>
  <ul>
  <li><p>There is clearly some learning occurring.</p>
  </li>
  <li><p>There is a strong random element throughout. As you say the epsilon is reduced to near zero by episode 2000, and I would assume a driving simulation is mostly deterministic but with a lot of state hidden from the agent, this mainly implies that episodes are started in different states.</p>
  </li>
  <li><p>There may be an effect from continued learning that causes the agent performance to vary so much. However, it is more likely towards the end that the agent is encountering new never-seen-before states and making poor decisions in them.</p>
  </li>
  <li><p>The resulting graph matches to my experience of training DQN agents where some of the hyperparameters are off. Probably you need to do some kind of exploration or search through those parameters.</p>
  </li>
  </ul>
  <p>From your comments:</p>
  <blockquote>
  <p>Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?</p>
  </blockquote>
  <p>These are possibilities that I cannot rule out by looking at the graph.</p>
  <p>However, I think it is more likely that you have some agent design or hyperparameters that are not a good fit to the problem. Those hyperparameters could be almost anything - not enough episodes to cover variation, neural network too simple, neural network too complicated, epsilon decay too fast, poor choice of regularisation, experience replay memory too small, poor choice of optimiser, etc etc.</p>
  <p>It is also possible you have limited the state model to the point that learning is hard or even impossible. A common beginner's mistake here is to use static sensor information such as a single current screenshot on each timestep, so the agent has no way to assess how fast it is going, or which way it is already turning etc (if your input includes direct knowledge of current speed, turning etc then this is probably not a problem for you, I am just raising one of many possibilities).</p>
  <p>One thing that may help you understand the agent's learning performance better is instead of plotting the <em>learning</em> episode graph, plot a <em>test</em> graph, perhaps once every ~100 episodes, where you run some number of episodes without learning and with <span class="math-container">$\epsilon = 0$</span>, and take average of the metrics you are interested in. This will remove some of the randomness from the values you are plotting and give you a much better sense of training progress than your current plot.</p>
  <p>With Q learning, you may also get significant benefit by setting a non-zero minimum <span class="math-container">$\epsilon$</span>, e.g. <span class="math-container">$\epsilon = \text{max}(\text{eps_decay}(n), 0.01)$</span> where <span class="math-container">$\text{eps_decay}(n)$</span> is your current epsilon decay function for step <span class="math-container">$n$</span>. That is because the off-policy updates with exploration are still helpful at all stages of learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-05T10:35:46.223Z
Answers.LastActivityDate: 2020-12-05T16:47:21.813Z
Answers.LastEditDate: 2020-12-05T16:47:21.813Z
Answers.UserDisplayName: Neil Slater
Body: <p>My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the <span class="math-container">$\epsilon$</span>-greedy stage, it still kills/crashes itself in subsequent episodes.</p>
  <p>I would like to know if this common in RL (D3QN) scenarios.</p>
  <p>A graph showing episodes vs steps has been placed below.</p>
  <p><a href="https://i.sstatic.net/ILqrC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ILqrC.png" alt="enter image description here" /></a></p>
  <p>As one can see, the agent reaches 400 steps in episode 1000. But, in the subsequent episode, it falls down below 50 steps.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-03T21:08:48.583Z
Comments.Text: Does your graph show steps during training?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-03T22:12:06.917Z
Comments.Text: Right, although the epsilon is gradually decreasing and therefore it is also exploiting instead of just exploring.
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T07:12:19.270Z
Comments.Text: For the graph, can you say what epsilon starts at for epsilon starts at for episode 1, and ends at for episode ~2200?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T11:24:41.210Z
Comments.Text: What agent are you using? Which environment (and maybe library)? Which hyper-parameters? I think you should provide also these details, because they may be useful.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-05T03:27:47.707Z
Comments.Text: @NeilSlater At episode 1, the epsilon should be 1. I do not know when it would have reached 0, although it reached there before episode 2200. Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-05T03:30:35.350Z
Comments.Text: @nbro The agent is a self driving vehicle. Regarding hyperparameters, I am not sure how describing them would help. 
  Honestly, this is my first independent RL project, therefore my questions might be a little silly :)
  Would adding the episode vs reward help?
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-05T03:32:51.937Z
Comments.Text: @desert_ranger Which RL model/algorithm are you using to guide the self-driving car? Well, in these cases, it may be useful to provide as much info as possible about your configuration, although I don't really know if it will turn out to be useful.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-05T04:30:52.453Z
Comments.Text: @nbro It is a slightly modified D3QN
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-05T16:47:49.300Z
Comments.Text: Thanks for accepting the answer. I have extended the answer a little with another key point that you may want to consider.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><epsilon-greedy-policy>
Title: Should my agent be taking varying number of steps?
UserDisplayName: desert_ranger
ThreadText: # Should my agent be taking varying number of steps?
  
  (Asked by: username_0 on 2020-12-03)
  
  My environment is set up so that my self-driving agent can take maximum of 400 steps (which is the end goal) before it resets with a completion reward. Despite attaining the end goal during the $\epsilon$-greedy stage, it still kills/crashes itself in subsequent episodes.
  
  I would like to know if this common in RL (D3QN) scenarios.
  
  A graph showing episodes vs steps has been placed below.
  
  [![enter image description here](https://i.sstatic.net/ILqrC.png)](https://i.sstatic.net/ILqrC.png)
  
  As one can see, the agent reaches 400 steps in episode 1000. But, in the subsequent episode, it falls down below 50 steps.
  
  **Tags:** reinforcement-learning, epsilon-greedy-policy
  
  > **Comments:**
  > **username_1**: Does your graph show steps during training?  
  > **username_0**: Right, although the epsilon is gradually decreasing and therefore it is also exploiting instead of just exploring.  
  > **username_1**: For the graph, can you say what epsilon starts at for epsilon starts at for episode 1, and ends at for episode ~2200?  
  > **username_2**: What agent are you using? Which environment (and maybe library)? Which hyper-parameters? I think you should provide also these details, because they may be useful.  
  > **username_0**: @username_1 At episode 1, the epsilon should be 1. I do not know when it would have reached 0, although it reached there before episode 2200. Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?  
  > **username_0**: @username_2 The agent is a self driving vehicle. Regarding hyperparameters, I am not sure how describing them would help.  Honestly, this is my first independent RL project, therefore my questions might be a little silly :) Would adding the episode vs reward help?  
  > **username_2**: @username_0 Which RL model/algorithm are you using to guide the self-driving car? Well, in these cases, it may be useful to provide as much info as possible about your configuration, although I don't really know if it will turn out to be useful.  
  > **username_0**: @username_2 It is a slightly modified D3QN  
  > **username_1**: Thanks for accepting the answer. I have extended the answer a little with another key point that you may want to consider.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-05. Score: 1 | Accepted answer
  
  Your graph looks to me like a typical learning curve plotted for training process in reinforcement learning.
  
  Looking at it in detail I can say:
  
  * There is clearly some learning occurring.
  * There is a strong random element throughout. As you say the epsilon is reduced to near zero by episode 2000, and I would assume a driving simulation is mostly deterministic but with a lot of state hidden from the agent, this mainly implies that episodes are started in different states.
  * There may be an effect from continued learning that causes the agent performance to vary so much. However, it is more likely towards the end that the agent is encountering new never-seen-before states and making poor decisions in them.
  * The resulting graph matches to my experience of training DQN agents where some of the hyperparameters are off. Probably you need to do some kind of exploration or search through those parameters.
  
  From your comments:
  
  > Perhaps the model is subsequently overfitting? Or maybe there is some catastrophic forgetting?
  
  These are possibilities that I cannot rule out by looking at the graph.
  
  However, I think it is more likely that you have some agent design or hyperparameters that are not a good fit to the problem. Those hyperparameters could be almost anything - not enough episodes to cover variation, neural network too simple, neural network too complicated, epsilon decay too fast, poor choice of regularisation, experience replay memory too small, poor choice of optimiser, etc etc.
  
  It is also possible you have limited the state model to the point that learning is hard or even impossible. A common beginner's mistake here is to use static sensor information such as a single current screenshot on each timestep, so the agent has no way to assess how fast it is going, or which way it is already turning etc (if your input includes direct knowledge of current speed, turning etc then this is probably not a problem for you, I am just raising one of many possibilities).
  
  One thing that may help you understand the agent's learning performance better is instead of plotting the *learning* episode graph, plot a *test* graph, perhaps once every ~100 episodes, where you run some number of episodes without learning and with $\epsilon = 0$, and take average of the metrics you are interested in. This will remove some of the randomness from the values you are plotting and give you a much better sense of training progress than your current plot.
  
  With Q learning, you may also get significant benefit by setting a non-zero minimum $\epsilon$, e.g. $\epsilon = \text{max}(\text{eps\_decay}(n), 0.01)$ where $\text{eps\_decay}(n)$ is your current epsilon decay function for step $n$. That is because the off-policy updates with exploration are still helpful at all stages of learning.
  
  
  ---
  
  