Answers.Body: <p>Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value <strong>V(s)</strong> function is estimated separately from the state-action value <strong>Q(s, a)</strong>. A new quantity, the advantage of an action, can then be defined as <strong>A(s, a) = Q(s, a) - V(s)</strong>. </p>
  
  <blockquote>
    <p>The Q function, however, measures the the value
    of choosing a particular action when in this state. The advantage
    function subtracts the value of the state from the Q
    function to obtain a relative measure of the importance of
    each action.</p>
    
    <p><a href="https://arxiv.org/pdf/1511.06581.pdf" rel="nofollow noreferrer">Dueling Network Architectures for Deep Reinforcement Learning</a></p>
  </blockquote>
  
  <hr>
  
  <p>To better direct you, here are 2 resources that could really help you understand why those differences are important.</p>
  
  <p><a href="https://medium.com/mlreview/speeding-up-dqn-on-pytorch-solving-pong-in-30-minutes-81a1bd2dff55" rel="nofollow noreferrer">Speeding up DQN on PyTorch: how to solve Pong in 30 minutes</a></p>
  
  <p>The main points of the blog are:</p>
  
  <ol>
  <li>Use a larger batch size and play several steps before updating</li>
  <li>Play and train in a separate process</li>
  <li>Use asyncronous cuda transfers</li>
  </ol>
  
  <p><a href="https://github.com/higgsfield/RL-Adventure" rel="nofollow noreferrer">RL Adventure</a></p>
  
  <p>This github library has easy to follow jupyter notebooks and links to all of the papers.
  It includes:</p>
  
  <ul>
  <li>DQN</li>
  <li>Double DQN</li>
  <li>Dueling DQN</li>
  <li>Prioritized Experience Replay</li>
  <li>Noise Networks for Exploration</li>
  <li>Distributional RL</li>
  <li>Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)</li>
  <li>Distributional RL with Quantile Regression</li>
  <li>Hierarchical Deep RL</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-30T15:40:59.627Z
Answers.Text: Thanks for your response. Are you telling me that this sort of performance, even anything close to it, is impossible using a normal DDQN?
Answers.UserDisplayName: hridayns
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-30T16:02:09.907Z
Answers.Text: Yep. Its not that a DDQN is a bad architecture. Its great and does what it was intended to do (i.e., prevent maximization bias (see section 6.7 in http://incompleteideas.net/book/bookdraft2017nov5.pdf)). But there exist qualities that an environment can posses which, when taken into consideration, allow a method to perform better within the same amount of computation time as methods which do not consider these qualities.
Answers.UserDisplayName: Jaden Travnik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-30T15:17:46.523Z
Answers.LastActivityDate: 2019-01-30T15:17:46.523Z
Answers.UserDisplayName: Jaden Travnik
Answers.Body: <p>Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).</p>
  
  <p>I made the following changes to my code after closely examining the OpenAI baselines code:</p>
  
  <ul>
  <li>Used <code>PongFrameskip-v4</code> instead of <code>PongDeterministic-v4</code></li>
  <li>Used a small replay buffer of size <strong>10000</strong></li>
  <li>During a <code>step_update()</code> or <code>replay()</code> call, changed the condition for a  <code>return</code> from <code>buffer_fill_size &lt; learn_start</code> to <code>t &lt; learn_start</code>, where <code>t</code> is the current timestep, <code>buffer_fill_size</code> is the current size of buffer that has been filled up with experience tuples, and <code>learn_start</code> is the number of timesteps to wait before starting to learn from the experience collected.</li>
  <li><p>Made sure that the <code>make_atari()</code> wrapper function is also called on the <code>env</code>:</p>
  
  <pre><code>ENV_GYM = 'PongFrameskip-v4'
  env = make_atari(ENV_GYM)
  env = wrap_deepmind(env, frame_stack=True, scale=False)
  </code></pre>
  
  <p>These wrappers may be implemented from scratch or can be obtained from the <a href="https://github.com/openai/baselines/blob/master/baselines/common/atari_wrappers.py" rel="nofollow noreferrer">OpenAI baseline Atari wrappers</a>. I personally used the latter since there is no point in reinventing the wheel.</p></li>
  </ul>
  
  <p><strong>Conclusion</strong>:</p>
  
  <p>The biggest step that I overlooked, or rather didn't pay much attention to was the input preprocessing. These few changes improved my DDQN from an average score saturation at -13 in almost 5000 episodes to +18 in about 700-800 episodes. That is indeed a huge difference. You can check out my implementation <a href="https://github.com/hridayns/Reinforcement-Learning/tree/master/Atari" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T10:58:39.387Z
Answers.Text: Could you mention specifically what was different between your original input processing and the improved version? Was this about scaling of NN inputs for instance?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T15:18:23.647Z
Answers.Text: By input processing, I mean the environment chosen (PongFrameskip-v4, instead of PongDeterministic-v4 in this case) and the wrapper methods `make_atari` and `wrap_deepmind`. I did not have the `make_atari` wrapper.
Answers.UserDisplayName: hridayns
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T03:56:15.743Z
Answers.LastActivityDate: 2019-02-10T03:56:15.743Z
Answers.UserDisplayName: hridayns
Body: <p><strong><em>Status</em></strong>:</p>
  
  <p>For a few weeks now, I have been working on a Double DQN agent for the <code>PongDeterministic-v4</code> environment, which you can find <a href="https://github.com/hridayns/Reinforcement-Learning/tree/master/Atari" rel="nofollow noreferrer">here</a>.</p>
  
  <p>A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.</p>
  
  <p><strong><em>Revelation</em></strong>:</p>
  
  <p>But then after cloning the OpenAI baselines <a href="https://github.com/openai/baselines" rel="nofollow noreferrer">repo</a>, I tried running <code>python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4</code>  and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.</p>
  
  <p><strong><em>Investigation</em></strong>:</p>
  
  <p>After going through the DQN baseline <a href="https://github.com/openai/baselines/tree/master/baselines/deepq" rel="nofollow noreferrer">code</a> by OpenAI, I was able to note a few differences:</p>
  
  <ul>
  <li>I use the <code>PongDeterministic-v4</code> environment but they use the <code>PongNoFrameskip-v4</code> environment </li>
  <li>I thought a larger <strong>replay buffer size</strong> was important, so I struggled (with the memory optimization) to ensure it was set to <strong>70000</strong> but they set it to a mere <strong>10000</strong>, and still got amazing results.</li>
  <li>I am using a normal <strong>Double DQN</strong>, but they seem to be using a <strong>Dueling Double DQN</strong>.</li>
  </ul>
  
  <p><strong><em>Results/Conclusion</em></strong></p>
  
  <p>I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone <strong>point me in the right direction</strong>?</p>
  
  <p>Any sort of help will be appreciated. Thanks!</p>
  
Tags: <deep-learning><reinforcement-learning><dqn><open-ai><deepmind>
Title: Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?
UserDisplayName: hridayns
ThreadText: # Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?
  
  (Asked by: username_0 on 2019-01-30)
  
  ***Status***:
  
  For a few weeks now, I have been working on a Double DQN agent for the `PongDeterministic-v4` environment, which you can find [here](https://github.com/username_0/Reinforcement-Learning/tree/master/Atari).
  
  A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.
  
  ***Revelation***:
  
  But then after cloning the OpenAI baselines [repo](https://github.com/openai/baselines), I tried running `python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4` and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.
  
  ***Investigation***:
  
  After going through the DQN baseline [code](https://github.com/openai/baselines/tree/master/baselines/deepq) by OpenAI, I was able to note a few differences:
  
  * I use the `PongDeterministic-v4` environment but they use the `PongNoFrameskip-v4` environment
  * I thought a larger **replay buffer size** was important, so I struggled (with the memory optimization) to ensure it was set to **70000** but they set it to a mere **10000**, and still got amazing results.
  * I am using a normal **Double DQN**, but they seem to be using a **Dueling Double DQN**.
  
  ***Results/Conclusion***
  
  I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone **point me in the right direction**?
  
  Any sort of help will be appreciated. Thanks!
  
  **Tags:** deep-learning, reinforcement-learning, dqn, open-ai, deepmind
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2019-02-10. Score: 3 | Accepted answer
  
  Although what @username_1 said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).
  
  I made the following changes to my code after closely examining the OpenAI baselines code:
  
  * Used `PongFrameskip-v4` instead of `PongDeterministic-v4`
  * Used a small replay buffer of size **10000**
  * During a `step_update()` or `replay()` call, changed the condition for a `return` from `buffer_fill_size < learn_start` to `t < learn_start`, where `t` is the current timestep, `buffer_fill_size` is the current size of buffer that has been filled up with experience tuples, and `learn_start` is the number of timesteps to wait before starting to learn from the experience collected.
  * Made sure that the `make_atari()` wrapper function is also called on the `env`:
  
    ```
    ENV_GYM = 'PongFrameskip-v4'
    env = make_atari(ENV_GYM)
    env = wrap_deepmind(env, frame_stack=True, scale=False)
  
    ```
  
    These wrappers may be implemented from scratch or can be obtained from the [OpenAI baseline Atari wrappers](https://github.com/openai/baselines/blob/master/baselines/common/atari_wrappers.py). I personally used the latter since there is no point in reinventing the wheel.
  
  **Conclusion**:
  
  The biggest step that I overlooked, or rather didn't pay much attention to was the input preprocessing. These few changes improved my DDQN from an average score saturation at -13 in almost 5000 episodes to +18 in about 700-800 episodes. That is indeed a huge difference. You can check out my implementation [here](https://github.com/username_0/Reinforcement-Learning/tree/master/Atari).
  
  > **Comments:**
  > **username_2**: Could you mention specifically what was different between your original input processing and the improved version? Was this about scaling of NN inputs for instance?  
  > **username_0**: By input processing, I mean the environment chosen (PongFrameskip-v4, instead of PongDeterministic-v4 in this case) and the wrapper methods `make_atari` and `wrap_deepmind`. I did not have the `make_atari` wrapper.  
  
  ---
  
  ### Answer by username_1 on 2019-01-30. Score: 3
  
  Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value **V(s)** function is estimated separately from the state-action value **Q(s, a)**. A new quantity, the advantage of an action, can then be defined as **A(s, a) = Q(s, a) - V(s)**.
  
  > The Q function, however, measures the the value
  > of choosing a particular action when in this state. The advantage
  > function subtracts the value of the state from the Q
  > function to obtain a relative measure of the importance of
  > each action.
  >
  > [Dueling Network Architectures for Deep Reinforcement Learning](https://arxiv.org/pdf/1511.06581.pdf)
  
  ---
  
  To better direct you, here are 2 resources that could really help you understand why those differences are important.
  
  [Speeding up DQN on PyTorch: how to solve Pong in 30 minutes](https://medium.com/mlreview/speeding-up-dqn-on-pytorch-solving-pong-in-30-minutes-81a1bd2dff55)
  
  The main points of the blog are:
  
  1. Use a larger batch size and play several steps before updating
  2. Play and train in a separate process
  3. Use asyncronous cuda transfers
  
  [RL Adventure](https://github.com/higgsfield/RL-Adventure)
  
  This github library has easy to follow jupyter notebooks and links to all of the papers.
  It includes:
  
  * DQN
  * Double DQN
  * Dueling DQN
  * Prioritized Experience Replay
  * Noise Networks for Exploration
  * Distributional RL
  * Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
  * Distributional RL with Quantile Regression
  * Hierarchical Deep RL
  
  > **Comments:**
  > **username_0**: Thanks for your response. Are you telling me that this sort of performance, even anything close to it, is impossible using a normal DDQN?  
  > **username_1**: Yep. Its not that a DDQN is a bad architecture. Its great and does what it was intended to do (i.e., prevent maximization bias (see section 6.7 in http://incompleteideas.net/book/bookdraft2017nov5.pdf)). But there exist qualities that an environment can posses which, when taken into consideration, allow a method to perform better within the same amount of computation time as methods which do not consider these qualities.  
  
  ---
  
  