<p>I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.</p>

<p>These lines should do the trick:</p>

<pre><code>self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
self.bias_mu = Parameter(torch.Tensor(out_features))
self.bias_sigma = Parameter(torch.Tensor(out_features))
</code></pre>

<p>In case you are not familiar, the <code>Parameter</code> class must be imported from <code>torch</code>:</p>

<pre><code>from torch.nn.parameter import Parameter
</code></pre>

CC BY-SA 4.0
2020-01-16T06:47:04.110Z
2020-01-16T06:47:04.110Z
Diego Gomez
<p>I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q value based on the encoded state.<br>
My <code>NoisyLinear</code> layers looks like this:</p>

<pre><code>class NoisyLinear(nn.Module):
  def __init__(self, in_features, out_features):
    super(NoisyLinear, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.sigma_zero = 0.5
    self.weight_mu = torch.empty(out_features, in_features)
    self.weight_sigma = torch.empty(out_features, in_features)
    self.weight_epsilon = torch.empty(out_features, in_features, requires_grad=False)
    self.bias_mu = torch.empty(out_features)
    self.bias_sigma = torch.empty(out_features)
    self.bias_epsilon = torch.empty(out_features, requires_grad=False)
    self.reset_parameters()
    self.reset_noise()

  def reset_parameters(self):
    mu_range = 1 / math.sqrt(self.in_features)
    self.weight_mu.data.uniform_(-mu_range, mu_range)
    self.weight_sigma.data.fill_(self.sigma_zero / math.sqrt(self.in_features))
    self.bias_mu.data.uniform_(-mu_range, mu_range)
    self.bias_sigma.data.fill_(self.sigma_zero / math.sqrt(self.out_features))

  def _scale_noise(self, size):
    x = torch.randn(size)
    return x.sign().mul_(x.abs().sqrt_())

  def reset_noise(self):
    epsilon_in = self._scale_noise(self.in_features)
    epsilon_out = self._scale_noise(self.out_features)
    self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
    self.bias_epsilon.copy_(epsilon_out)

  def forward(self, input):
    return F.linear(input, self.weight_mu + self.weight_sigma * self.weight_epsilon, self.bias_mu + self.bias_sigma * self.bias_epsilon)
</code></pre>

<p>However, with the default hyperparameters from the (sigma_0 = 0.5), the agent does not explore at all, and even if I crank it up to sigma_0 = 5, it works way worse than epsilon greedy.<br>
(When I use noisy nets I don't use epsilon greedy).</p>

<reinforcement-learning><dqn>
NoisyNet DQN with default parameters not exploring
Luca Thiede
# NoisyNet DQN with default parameters not exploring

(Asked by: username_0 on 2020-01-14)

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q value based on the encoded state.
My NoisyLinear layers looks like this:
class NoisyLinear(nn.Module):
  def __init__(self, in_features, out_features):
    super(NoisyLinear, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.sigma_zero = 0.5
    self.weight_mu = torch.empty(out_features, in_features)
    self.weight_sigma = torch.empty(out_features, in_features)
    self.weight_epsilon = torch.empty(out_features, in_features, requires_grad=False)
    self.bias_mu = torch.empty(out_features)
    self.bias_sigma = torch.empty(out_features)
    self.bias_epsilon = torch.empty(out_features, requires_grad=False)
    self.reset_parameters()
    self.reset_noise()

  def reset_parameters(self):
    mu_range = 1 / math.sqrt(self.in_features)
    self.weight_mu.data.uniform_(-mu_range, mu_range)
    self.weight_sigma.data.fill_(self.sigma_zero / math.sqrt(self.in_features))
    self.bias_mu.data.uniform_(-mu_range, mu_range)
    self.bias_sigma.data.fill_(self.sigma_zero / math.sqrt(self.out_features))

  def _scale_noise(self, size):
    x = torch.randn(size)
    return x.sign().mul_(x.abs().sqrt_())

  def reset_noise(self):
    epsilon_in = self._scale_noise(self.in_features)
    epsilon_out = self._scale_noise(self.out_features)
    self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
    self.bias_epsilon.copy_(epsilon_out)

  def forward(self, input):
    return F.linear(input, self.weight_mu + self.weight_sigma * self.weight_epsilon, self.bias_mu + self.bias_sigma * self.bias_epsilon)

However, with the default hyperparameters from the (sigma_0 = 0.5), the agent does not explore at all, and even if I crank it up to sigma_0 = 5, it works way worse than epsilon greedy.
(When I use noisy nets I don't use epsilon greedy).

**Tags:** reinforcement-learning, dqn


---

## 1 answers

### Answer by username_1 on 2020-01-16. Score: 0

I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.
These lines should do the trick:
self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
self.bias_mu = Parameter(torch.Tensor(out_features))
self.bias_sigma = Parameter(torch.Tensor(out_features))

In case you are not familiar, the Parameter class must be imported from torch:
from torch.nn.parameter import Parameter


---

