<p>Here is my <a href="https://colab.research.google.com/drive/1ca3SZtZFhqeBTiuv81_Mn9qIdZ5vg6fF#scrollTo=9ctKD177RTPa" rel="nofollow noreferrer">code</a></p>

<p>Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method which may help me improving the convergence speed. I found the following article : <a href="https://arxiv.org/pdf/1706.10295v3.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1706.10295v3.pdf</a>. It says I can use an Independent Gaussian Noise to outperform an standard DQN. </p>

<p>Here is my Noisy DQN model : </p>

<pre><code>import math
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np


class NoisyLinear(nn.Linear): #Independent Gaussian Noise used with NoisyDQN model
    def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
        super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)

        self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
        self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))

        if bias: 
            self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
            self.register_buffer("epsilon_bias", torch.zeros(out_features))

        self.reset_parameters()

    def reset_parameters(self):
        std = math.sqrt(3/self.in_features)
        self.weight.data.uniform_(-std, std)
        self.bias.data.uniform_(-std, std)

    def forward(self, input):
        self.epsilon_weight.normal_()
        bias = self.bias
        if bias is not None:
            self.epsilon_bias.normal_()
            bias = bias + self.sigma_bias * self.epsilon_bias.data
        return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight.data, bias)


class NoisyDQN(nn.Module):
    """
    Look at https://arxiv.org/pdf/1706.10295v3.pdf
    """

    def __init__(self, input_shape, num_actions):
        super(NoisyDQN, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(in_channels=input_shape[0], out_channels=32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
            nn.ReLU(),
        )

        self.conv_output_size = self.get_output_size(input_shape)

        self.linear = nn.Sequential(
           NoisyLinear(in_features=self.conv_output_size, out_features=512),
           nn.ReLU(),
           NoisyLinear(in_features=512, out_features=num_actions)
        )

    def get_output_size(self, input_shape):
        output = self.conv(torch.zeros((1, *input_shape)))
        return int(np.prod(output.shape))

    def forward(self, input):
        self.layer1 = self.conv(input)
        self.layer1 = self.layer1.reshape(-1, self.conv_output_size)

        return self.linear(self.layer1)
</code></pre>

<p>The idea is to replace the epsilon greedy action selection and my standard DQN model by the Noisy Network you can see just above.</p>

<p>The code run successfully, but it doesn't improve even a bit. How can I fix that?</p>

<p><strong>UPDATE</strong></p>

<p><a href="https://i.sstatic.net/DgrcY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DgrcY.png" alt="enter image description here"></a></p>

<p>After nearly 200k episodes, I am still between <code>1.5</code> and <code>2</code> reward. The maximum reward I can get on the Atari Breakout game is about <code>500</code>. With a standard DQN, after 100k episodes, I am near <code>11</code> as reward.</p>

<p>On the above picture the X-axis is the number of episodes and the Y-axis is the mean reward over the last 100 rewards. The Y-axis is describe with 
<code>mean_reward = np.mean(self.total_rewards[-100:])</code></p>

<p><strong>UPDATE</strong></p>

<p>After about 8 hours of training, I got this</p>

<p><a href="https://i.sstatic.net/v29Dt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v29Dt.png" alt="enter image description here"></a></p>

<p>As you can see, it is not working as good as in the paper. I worked a lot with the hyperparameters, but not changed. </p>

CC BY-SA 4.0
2020-04-17T18:23:07.820Z
Can you clarify what you mean by "it doesn't improve even a bit"? Do you mean that by using this Noisy Network you don't get rewards? Maybe provide a plot of the behaviour of your agent.
nbro
CC BY-SA 4.0
2020-04-17T18:37:20.553Z
@nbro I will add more info very soon. I am waiting to have a clear plot.
jgauth
CC BY-SA 4.0
2020-04-17T20:34:11.460Z
Without even knowing how noisy network work, I suggest you reduce the variance (where you use it) or, in general, noisy that you use during your training process. For example, I see you're doing `std = math.sqrt(3/self.in_features)`. Isn't `std` maybe too high?
nbro
CC BY-SA 4.0
2020-04-17T20:42:37.553Z
Hm, that seems like a reasonable range to initialize the parameters.
nbro
CC BY-SA 4.0
2020-04-17T20:43:23.820Z
@nbro `self.in_features` is equal `3136`. How it could be high? Sorry, I got wrong.
jgauth
CC BY-SA 4.0
2020-04-17T20:44:34.203Z
You're passing `self.conv_output_size` to it. See how `self.get_output_size` computes `self.conv_output_size`. What's your `input_shape`?
nbro
CC BY-SA 4.0
2020-04-17T20:51:00.020Z
@nbro The `input_shape` is `(4, 84, 84)`.  `self.get_output_size` is used just to flatten the size between the conv layers and the linear layers.  I inspected the result and it seems to be fine
jgauth
CC BY-SA 4.0
2020-04-17T20:56:59.790Z
Anyway, the fact that `self.in_features` is 3136 will restrict the range of `self.weight` and `self.bias`. Is that ok? Honestly, I would need to look at the paper that introduced this noisy networks in order to understand better what's going on.
nbro
CC BY-SA 4.0
2020-04-17T21:00:34.077Z
@Normally, what you see in NoisyDQN reflects what you see in the paper regarding the Independant Gaussian Noise.
jgauth
CC BY-SA 4.0
2020-04-17T21:31:50.673Z
@nbro I played with `self.reset_parameters()` and it changed nothing really. I have deleted and changed the values of `std`. I got the same patterns
jgauth
CC BY-SA 4.0
2020-04-17T21:40:07.903Z
What about the `forward` method of `NoisyLinear`? Are you sure it follows the instructions in the paper?
nbro
CC BY-SA 4.0
2020-04-17T22:02:33.447Z
@Yeah, I am sure. It is probably not that different from the Gaussian Noise you already know.
jgauth
CC BY-SA 4.0
2020-04-17T22:17:16.470Z
@nbro I think I could make a modification on `forward()` method from class `NoisyDQN`. I might make a good difference.
jgauth
CC BY-SA 4.0
2020-04-17T22:26:32.640Z
@nbro No, I am still having the same problem. Both forward() method look fine to me
jgauth
<reinforcement-learning><python><dqn><pytorch>
Replace epsilon greedy action selection and the standard DQN by an Independent Gaussian Noise Network Model
jgauth
# Replace epsilon greedy action selection and the standard DQN by an Independent Gaussian Noise Network Model

(Asked by: username_0 on 2020-04-17)

Here is my code
Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method which may help me improving the convergence speed. I found the following article : https://arxiv.org/pdf/1706.10295v3.pdf. It says I can use an Independent Gaussian Noise to outperform an standard DQN. 
Here is my Noisy DQN model : 
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np


class NoisyLinear(nn.Linear): #Independent Gaussian Noise used with NoisyDQN model
    def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
        super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)

        self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
        self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))

        if bias: 
            self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
            self.register_buffer("epsilon_bias", torch.zeros(out_features))

        self.reset_parameters()

    def reset_parameters(self):
        std = math.sqrt(3/self.in_features)
        self.weight.data.uniform_(-std, std)
        self.bias.data.uniform_(-std, std)

    def forward(self, input):
        self.epsilon_weight.normal_()
        bias = self.bias
        if bias is not None:
            self.epsilon_bias.normal_()
            bias = bias + self.sigma_bias * self.epsilon_bias.data
        return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight.data, bias)


class NoisyDQN(nn.Module):
    """
    Look at https://arxiv.org/pdf/1706.10295v3.pdf
    """

    def __init__(self, input_shape, num_actions):
        super(NoisyDQN, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(in_channels=input_shape[0], out_channels=32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
            nn.ReLU(),
        )

        self.conv_output_size = self.get_output_size(input_shape)

        self.linear = nn.Sequential(
           NoisyLinear(in_features=self.conv_output_size, out_features=512),
           nn.ReLU(),
           NoisyLinear(in_features=512, out_features=num_actions)
        )

    def get_output_size(self, input_shape):
        output = self.conv(torch.zeros((1, *input_shape)))
        return int(np.prod(output.shape))

    def forward(self, input):
        self.layer1 = self.conv(input)
        self.layer1 = self.layer1.reshape(-1, self.conv_output_size)

        return self.linear(self.layer1)

The idea is to replace the epsilon greedy action selection and my standard DQN model by the Noisy Network you can see just above.
The code run successfully, but it doesn't improve even a bit. How can I fix that?
UPDATE

After nearly 200k episodes, I am still between 1.5 and 2 reward. The maximum reward I can get on the Atari Breakout game is about 500. With a standard DQN, after 100k episodes, I am near 11 as reward.
On the above picture the X-axis is the number of episodes and the Y-axis is the mean reward over the last 100 rewards. The Y-axis is describe with 
mean_reward = np.mean(self.total_rewards[-100:])
UPDATE
After about 8 hours of training, I got this

As you can see, it is not working as good as in the paper. I worked a lot with the hyperparameters, but not changed.

**Tags:** reinforcement-learning, python, dqn, pytorch

> **Comments:**
> **username_1**: Can you clarify what you mean by "it doesn't improve even a bit"? Do you mean that by using this Noisy Network you don't get rewards? Maybe provide a plot of the behaviour of your agent.  
> **username_0**: @username_1 I will add more info very soon. I am waiting to have a clear plot.  
> **username_1**: Without even knowing how noisy network work, I suggest you reduce the variance (where you use it) or, in general, noisy that you use during your training process. For example, I see you're doing `std = math.sqrt(3/self.in_features)`. Isn't `std` maybe too high?  
> **username_1**: Hm, that seems like a reasonable range to initialize the parameters.  
> **username_0**: @username_1 `self.in_features` is equal `3136`. How it could be high? Sorry, I got wrong.  
> **username_1**: You're passing `self.conv_output_size` to it. See how `self.get_output_size` computes `self.conv_output_size`. What's your `input_shape`?  
> **username_0**: @username_1 The `input_shape` is `(4, 84, 84)`.  `self.get_output_size` is used just to flatten the size between the conv layers and the linear layers.  I inspected the result and it seems to be fine  
> **username_1**: Anyway, the fact that `self.in_features` is 3136 will restrict the range of `self.weight` and `self.bias`. Is that ok? Honestly, I would need to look at the paper that introduced this noisy networks in order to understand better what's going on.  
> **username_0**: @Normally, what you see in NoisyDQN reflects what you see in the paper regarding the Independant Gaussian Noise.  
> **username_0**: @username_1 I played with `self.reset_parameters()` and it changed nothing really. I have deleted and changed the values of `std`. I got the same patterns  
> **username_1**: What about the `forward` method of `NoisyLinear`? Are you sure it follows the instructions in the paper?  
> **username_0**: @Yeah, I am sure. It is probably not that different from the Gaussian Noise you already know.  
> **username_0**: @username_1 I think I could make a modification on `forward()` method from class `NoisyDQN`. I might make a good difference.  
> **username_0**: @username_1 No, I am still having the same problem. Both forward() method look fine to me  

---

## 0 answers

