Body: <p>Here is my <a href="https://colab.research.google.com/drive/1ca3SZtZFhqeBTiuv81_Mn9qIdZ5vg6fF#scrollTo=9ctKD177RTPa" rel="nofollow noreferrer">code</a></p>
  
  <p>Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method which may help me improving the convergence speed. I found the following article : <a href="https://arxiv.org/pdf/1706.10295v3.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1706.10295v3.pdf</a>. It says I can use an Independent Gaussian Noise to outperform an standard DQN. </p>
  
  <p>Here is my Noisy DQN model : </p>
  
  <pre><code>import math
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import numpy as np
  
  
  class NoisyLinear(nn.Linear): #Independent Gaussian Noise used with NoisyDQN model
      def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
          super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)
  
          self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
          self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))
  
          if bias: 
              self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
              self.register_buffer("epsilon_bias", torch.zeros(out_features))
  
          self.reset_parameters()
  
      def reset_parameters(self):
          std = math.sqrt(3/self.in_features)
          self.weight.data.uniform_(-std, std)
          self.bias.data.uniform_(-std, std)
  
      def forward(self, input):
          self.epsilon_weight.normal_()
          bias = self.bias
          if bias is not None:
              self.epsilon_bias.normal_()
              bias = bias + self.sigma_bias * self.epsilon_bias.data
          return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight.data, bias)
  
  
  class NoisyDQN(nn.Module):
      """
      Look at https://arxiv.org/pdf/1706.10295v3.pdf
      """
  
      def __init__(self, input_shape, num_actions):
          super(NoisyDQN, self).__init__()
  
          self.conv = nn.Sequential(
              nn.Conv2d(in_channels=input_shape[0], out_channels=32, kernel_size=8, stride=4),
              nn.ReLU(),
              nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
              nn.ReLU(),
              nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
              nn.ReLU(),
          )
  
          self.conv_output_size = self.get_output_size(input_shape)
  
          self.linear = nn.Sequential(
             NoisyLinear(in_features=self.conv_output_size, out_features=512),
             nn.ReLU(),
             NoisyLinear(in_features=512, out_features=num_actions)
          )
  
      def get_output_size(self, input_shape):
          output = self.conv(torch.zeros((1, *input_shape)))
          return int(np.prod(output.shape))
  
      def forward(self, input):
          self.layer1 = self.conv(input)
          self.layer1 = self.layer1.reshape(-1, self.conv_output_size)
  
          return self.linear(self.layer1)
  </code></pre>
  
  <p>The idea is to replace the epsilon greedy action selection and my standard DQN model by the Noisy Network you can see just above.</p>
  
  <p>The code run successfully, but it doesn't improve even a bit. How can I fix that?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p><a href="https://i.sstatic.net/DgrcY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DgrcY.png" alt="enter image description here"></a></p>
  
  <p>After nearly 200k episodes, I am still between <code>1.5</code> and <code>2</code> reward. The maximum reward I can get on the Atari Breakout game is about <code>500</code>. With a standard DQN, after 100k episodes, I am near <code>11</code> as reward.</p>
  
  <p>On the above picture the X-axis is the number of episodes and the Y-axis is the mean reward over the last 100 rewards. The Y-axis is describe with 
  <code>mean_reward = np.mean(self.total_rewards[-100:])</code></p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>After about 8 hours of training, I got this</p>
  
  <p><a href="https://i.sstatic.net/v29Dt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v29Dt.png" alt="enter image description here"></a></p>
  
  <p>As you can see, it is not working as good as in the paper. I worked a lot with the hyperparameters, but not changed. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T18:23:07.820Z
Comments.Text: Can you clarify what you mean by "it doesn't improve even a bit"? Do you mean that by using this Noisy Network you don't get rewards? Maybe provide a plot of the behaviour of your agent.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T18:37:20.553Z
Comments.Text: @nbro I will add more info very soon. I am waiting to have a clear plot.
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T20:34:11.460Z
Comments.Text: Without even knowing how noisy network work, I suggest you reduce the variance (where you use it) or, in general, noisy that you use during your training process. For example, I see you're doing `std = math.sqrt(3/self.in_features)`. Isn't `std` maybe too high?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T20:42:37.553Z
Comments.Text: Hm, that seems like a reasonable range to initialize the parameters.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T20:43:23.820Z
Comments.Text: @nbro `self.in_features` is equal `3136`. How it could be high? Sorry, I got wrong.
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T20:44:34.203Z
Comments.Text: You're passing `self.conv_output_size` to it. See how `self.get_output_size` computes `self.conv_output_size`. What's your `input_shape`?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T20:51:00.020Z
Comments.Text: @nbro The `input_shape` is `(4, 84, 84)`.  `self.get_output_size` is used just to flatten the size between the conv layers and the linear layers.  I inspected the result and it seems to be fine
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T20:56:59.790Z
Comments.Text: Anyway, the fact that `self.in_features` is 3136 will restrict the range of `self.weight` and `self.bias`. Is that ok? Honestly, I would need to look at the paper that introduced this noisy networks in order to understand better what's going on.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T21:00:34.077Z
Comments.Text: @Normally, what you see in NoisyDQN reflects what you see in the paper regarding the Independant Gaussian Noise.
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T21:31:50.673Z
Comments.Text: @nbro I played with `self.reset_parameters()` and it changed nothing really. I have deleted and changed the values of `std`. I got the same patterns
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T21:40:07.903Z
Comments.Text: What about the `forward` method of `NoisyLinear`? Are you sure it follows the instructions in the paper?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T22:02:33.447Z
Comments.Text: @Yeah, I am sure. It is probably not that different from the Gaussian Noise you already know.
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T22:17:16.470Z
Comments.Text: @nbro I think I could make a modification on `forward()` method from class `NoisyDQN`. I might make a good difference.
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T22:26:32.640Z
Comments.Text: @nbro No, I am still having the same problem. Both forward() method look fine to me
Comments.UserDisplayName: jgauth
Tags: <reinforcement-learning><python><dqn><pytorch>
Title: Replace epsilon greedy action selection and the standard DQN by an Independent Gaussian Noise Network Model
UserDisplayName: jgauth
ThreadText: # Replace epsilon greedy action selection and the standard DQN by an Independent Gaussian Noise Network Model
  
  (Asked by: username_0 on 2020-04-17)
  
  Here is my [code](https://colab.research.google.com/drive/1ca3SZtZFhqeBTiuv81_Mn9qIdZ5vg6fF#scrollTo=9ctKD177RTPa)
  
  Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method which may help me improving the convergence speed. I found the following article : <https://arxiv.org/pdf/1706.10295v3.pdf>. It says I can use an Independent Gaussian Noise to outperform an standard DQN.
  
  Here is my Noisy DQN model :
  
  ```
  import math
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import numpy as np
  
  class NoisyLinear(nn.Linear): #Independent Gaussian Noise used with NoisyDQN model
      def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
          super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)
  
          self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
          self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))
  
          if bias: 
              self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
              self.register_buffer("epsilon_bias", torch.zeros(out_features))
  
          self.reset_parameters()
  
      def reset_parameters(self):
          std = math.sqrt(3/self.in_features)
          self.weight.data.uniform_(-std, std)
          self.bias.data.uniform_(-std, std)
  
      def forward(self, input):
          self.epsilon_weight.normal_()
          bias = self.bias
          if bias is not None:
              self.epsilon_bias.normal_()
              bias = bias + self.sigma_bias * self.epsilon_bias.data
          return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight.data, bias)
  
  class NoisyDQN(nn.Module):
      """
      Look at https://arxiv.org/pdf/1706.10295v3.pdf
      """
  
      def __init__(self, input_shape, num_actions):
          super(NoisyDQN, self).__init__()
  
          self.conv = nn.Sequential(
              nn.Conv2d(in_channels=input_shape[0], out_channels=32, kernel_size=8, stride=4),
              nn.ReLU(),
              nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
              nn.ReLU(),
              nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
              nn.ReLU(),
          )
  
          self.conv_output_size = self.get_output_size(input_shape)
  
          self.linear = nn.Sequential(
             NoisyLinear(in_features=self.conv_output_size, out_features=512),
             nn.ReLU(),
             NoisyLinear(in_features=512, out_features=num_actions)
          )
  
      def get_output_size(self, input_shape):
          output = self.conv(torch.zeros((1, *input_shape)))
          return int(np.prod(output.shape))
  
      def forward(self, input):
          self.layer1 = self.conv(input)
          self.layer1 = self.layer1.reshape(-1, self.conv_output_size)
  
          return self.linear(self.layer1)
  
  ```
  
  The idea is to replace the epsilon greedy action selection and my standard DQN model by the Noisy Network you can see just above.
  
  The code run successfully, but it doesn't improve even a bit. How can I fix that?
  
  **UPDATE**
  
  [![enter image description here](https://i.sstatic.net/DgrcY.png)](https://i.sstatic.net/DgrcY.png)
  
  After nearly 200k episodes, I am still between `1.5` and `2` reward. The maximum reward I can get on the Atari Breakout game is about `500`. With a standard DQN, after 100k episodes, I am near `11` as reward.
  
  On the above picture the X-axis is the number of episodes and the Y-axis is the mean reward over the last 100 rewards. The Y-axis is describe with
  `mean_reward = np.mean(self.total_rewards[-100:])`
  
  **UPDATE**
  
  After about 8 hours of training, I got this
  
  [![enter image description here](https://i.sstatic.net/v29Dt.png)](https://i.sstatic.net/v29Dt.png)
  
  As you can see, it is not working as good as in the paper. I worked a lot with the hyperparameters, but not changed.
  
  **Tags:** reinforcement-learning, python, dqn, pytorch
  
  > **Comments:**
  > **username_1**: Can you clarify what you mean by "it doesn't improve even a bit"? Do you mean that by using this Noisy Network you don't get rewards? Maybe provide a plot of the behaviour of your agent.  
  > **username_0**: @username_1 I will add more info very soon. I am waiting to have a clear plot.  
  > **username_1**: Without even knowing how noisy network work, I suggest you reduce the variance (where you use it) or, in general, noisy that you use during your training process. For example, I see you're doing `std = math.sqrt(3/self.in_features)`. Isn't `std` maybe too high?  
  > **username_1**: Hm, that seems like a reasonable range to initialize the parameters.  
  > **username_0**: @username_1 `self.in_features` is equal `3136`. How it could be high? Sorry, I got wrong.  
  > **username_1**: You're passing `self.conv_output_size` to it. See how `self.get_output_size` computes `self.conv_output_size`. What's your `input_shape`?  
  > **username_0**: @username_1 The `input_shape` is `(4, 84, 84)`.  `self.get_output_size` is used just to flatten the size between the conv layers and the linear layers.  I inspected the result and it seems to be fine  
  > **username_1**: Anyway, the fact that `self.in_features` is 3136 will restrict the range of `self.weight` and `self.bias`. Is that ok? Honestly, I would need to look at the paper that introduced this noisy networks in order to understand better what's going on.  
  > **username_0**: @Normally, what you see in NoisyDQN reflects what you see in the paper regarding the Independant Gaussian Noise.  
  > **username_0**: @username_1 I played with `self.reset_parameters()` and it changed nothing really. I have deleted and changed the values of `std`. I got the same patterns  
  > **username_1**: What about the `forward` method of `NoisyLinear`? Are you sure it follows the instructions in the paper?  
  > **username_0**: @Yeah, I am sure. It is probably not that different from the Gaussian Noise you already know.  
  > **username_0**: @username_1 I think I could make a modification on `forward()` method from class `NoisyDQN`. I might make a good difference.  
  > **username_0**: @username_1 No, I am still having the same problem. Both forward() method look fine to me  
  
  ---
  
  ## 0 answers
  
  