Answers.Body: <p>I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing.</p>
  <h2>More exploration in long term</h2>
  <p>I think you have correctly identified that exploration could be an issue. In off-policy learning (which Q-learning is an instance of), it is usual to set a minimum exploration rate. It is a hyperparameter that you need to manage. Set too high, the agent will never experience the best rewards as it will make too many mistakes. Set too low, the agent will not explore enough to find the correct alternative actions when the opportunity to learn them occurs.</p>
  <p>I would suggest for you something like:</p>
  <pre><code>epsilon = max(min(1.0, 1000 / (frames + 1)), 0.01)
  </code></pre>
  <p>You can choose numbers other than 0.01, but I think that is a reasonable start for many Atari games. You could try higher, up to 0.1 in games which are more forgiving of mistakes.</p>
  <h2>Remove dropout</h2>
  <p>I am not sure why, but I always have problems with dropout in RL neural networks. Try removing the dropout layer.</p>
  <h2>More convolutional layers</h2>
  <p>Convolutional layers are very efficient generalisers for vision and grid-based problems. You won't really benefit much from having a single layer though. I would add another two, increase the number of output channels.</p>
  <h2>Maybe state representation?</h2>
  <p>It is not clear from your description whether you are using a single colour frame for the state representation, or stacked greyscale frames for the last 3 inputs. It should be the latter, and if you want to more closely replicate the orginal DQN Atari paper, you should take the previous 4 frames as input.</p>
  <p>In addition, you should be normalising the input into range <span class="math-container">$[0,1]$</span> or <span class="math-container">$[-1,1]$</span>. The native image range <span class="math-container">$[0,255]$</span> is tricky for neural networks to process, and quite common for value functions to get stuck if you don't normalise.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T15:26:02.543Z
Answers.Text: This really helped, thanks! The reward graph looks much more normal now. It more greatly resembles what I see in the paper and related works.
Answers.UserDisplayName: user192361237
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T15:37:58.457Z
Answers.Text: @RyanRudes: No problem. Which parts out of interest did you need to change?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T15:39:59.247Z
Answers.Text: I also just realized that the paper had trained the agent for 5 million frames for most of their graphs. Looking at the charts in the paper, reward didn’t even rise significantly until 500,000 frames in, and I hadn’t even trained up to there yet. So I severely underestimated the amount of training these require.
Answers.UserDisplayName: user192361237
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T15:48:50.500Z
Answers.Text: I took into account all of your suggestions (I already had the normalization implemented though). They were very helpful, thanks.
Answers.UserDisplayName: user192361237
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T10:28:08.677Z
Answers.LastActivityDate: 2020-08-28T10:28:08.677Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but none about the environment. It has learned that a good immediate strategy is to simply jump down both diagonals and fall of the board, thus completing a large portion of the first level. However, it remains to understand neither the boundaries of the board to prevent jumping off, nor anything about avoiding enemies. I’m asking this here, instead of Stack Overflow because it is a more general question with less of a need in terms of programming understanding. Simply, I am asking whether or not this is a matter of a pore exploration policy (which I presume). If you agree, what should be a better exploration policy for Q*bert that would facilitate my agent’s learning experience?</p>
  <p>As per the request of a comment:</p>
  <blockquote>
  <p>Could you add what your current exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.</p>
  </blockquote>
  <p>Here are my parameters:</p>
  <ul>
  <li>Exploration policy: epsilon = <code>min(1.0, 1000 / (frames + 1))</code></li>
  <li>Replay Memory = 20,000 frames</li>
  <li>Batch size = 32 transitions</li>
  <li>NN architecture: Conv2D(64, 3, 2), Dropout(0.2), Dense(32, relu), Dense(32, relu), Dense(num_actions, linear)</li>
  <li>Steps per target network copy: 100</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-27T20:24:46.643Z
Comments.Text: Could you add what your *current* exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-27T20:43:09.857Z
Comments.Text: I’ve added the information you requested.
Comments.UserDisplayName: user192361237
Tags: <reinforcement-learning><q-learning><dqn><deep-rl><exploration-exploitation-tradeoff>
Title: What is the optimal exploration-exploitation trade-off in Q*bert?
UserDisplayName: user192361237
ThreadText: # What is the optimal exploration-exploitation trade-off in Q*bert?
  
  (Asked by: username_0 on 2020-08-27)
  
  I am training an RL agent with Deep Q-learning + Experience Replay on the Q\*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but none about the environment. It has learned that a good immediate strategy is to simply jump down both diagonals and fall of the board, thus completing a large portion of the first level. However, it remains to understand neither the boundaries of the board to prevent jumping off, nor anything about avoiding enemies. I’m asking this here, instead of Stack Overflow because it is a more general question with less of a need in terms of programming understanding. Simply, I am asking whether or not this is a matter of a pore exploration policy (which I presume). If you agree, what should be a better exploration policy for Q\*bert that would facilitate my agent’s learning experience?
  
  As per the request of a comment:
  
  > Could you add what your current exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.
  
  Here are my parameters:
  
  * Exploration policy: epsilon = `min(1.0, 1000 / (frames + 1))`
  * Replay Memory = 20,000 frames
  * Batch size = 32 transitions
  * NN architecture: Conv2D(64, 3, 2), Dropout(0.2), Dense(32, relu), Dense(32, relu), Dense(num\_actions, linear)
  * Steps per target network copy: 100
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl, exploration-exploitation-tradeoff
  > **Comments:**
  > **username_1**: Could you add what your *current* exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.  
  > **username_0**: I’ve added the information you requested.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-28. Score: 1 | Accepted answer
  
  I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing.
  More exploration in long term
  I think you have correctly identified that exploration could be an issue. In off-policy learning (which Q-learning is an instance of), it is usual to set a minimum exploration rate. It is a hyperparameter that you need to manage. Set too high, the agent will never experience the best rewards as it will make too many mistakes. Set too low, the agent will not explore enough to find the correct alternative actions when the opportunity to learn them occurs.
  I would suggest for you something like:
  epsilon = max(min(1.0, 1000 / (frames + 1)), 0.01)
  
  You can choose numbers other than 0.01, but I think that is a reasonable start for many Atari games. You could try higher, up to 0.1 in games which are more forgiving of mistakes.
  Remove dropout
  I am not sure why, but I always have problems with dropout in RL neural networks. Try removing the dropout layer.
  More convolutional layers
  Convolutional layers are very efficient generalisers for vision and grid-based problems. You won't really benefit much from having a single layer though. I would add another two, increase the number of output channels.
  Maybe state representation?
  It is not clear from your description whether you are using a single colour frame for the state representation, or stacked greyscale frames for the last 3 inputs. It should be the latter, and if you want to more closely replicate the orginal DQN Atari paper, you should take the previous 4 frames as input.
  In addition, you should be normalising the input into range $[0,1]$ or $[-1,1]$. The native image range $[0,255]$ is tricky for neural networks to process, and quite common for value functions to get stuck if you don't normalise.
  
  > **Comments:**
  > **username_0**: This really helped, thanks! The reward graph looks much more normal now. It more greatly resembles what I see in the paper and related works.  
  > **username_1**: @RyanRudes: No problem. Which parts out of interest did you need to change?  
  > **username_0**: I also just realized that the paper had trained the agent for 5 million frames for most of their graphs. Looking at the charts in the paper, reward didn’t even rise significantly until 500,000 frames in, and I hadn’t even trained up to there yet. So I severely underestimated the amount of training these require.  
  > **username_0**: I took into account all of your suggestions (I already had the normalization implemented though). They were very helpful, thanks.  
  
  ---
  
  