Answers.Body: <p>I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.</p>
  
  <p>These lines should do the trick:</p>
  
  <pre><code>self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
  self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
  self.bias_mu = Parameter(torch.Tensor(out_features))
  self.bias_sigma = Parameter(torch.Tensor(out_features))
  </code></pre>
  
  <p>In case you are not familiar, the <code>Parameter</code> class must be imported from <code>torch</code>:</p>
  
  <pre><code>from torch.nn.parameter import Parameter
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-16T06:47:04.110Z
Answers.LastActivityDate: 2020-01-16T06:47:04.110Z
Answers.UserDisplayName: Diego Gomez
Body: <p>I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q value based on the encoded state.<br>
  My <code>NoisyLinear</code> layers looks like this:</p>
  
  <pre><code>class NoisyLinear(nn.Module):
    def __init__(self, in_features, out_features):
      super(NoisyLinear, self).__init__()
      self.in_features = in_features
      self.out_features = out_features
      self.sigma_zero = 0.5
      self.weight_mu = torch.empty(out_features, in_features)
      self.weight_sigma = torch.empty(out_features, in_features)
      self.weight_epsilon = torch.empty(out_features, in_features, requires_grad=False)
      self.bias_mu = torch.empty(out_features)
      self.bias_sigma = torch.empty(out_features)
      self.bias_epsilon = torch.empty(out_features, requires_grad=False)
      self.reset_parameters()
      self.reset_noise()
  
    def reset_parameters(self):
      mu_range = 1 / math.sqrt(self.in_features)
      self.weight_mu.data.uniform_(-mu_range, mu_range)
      self.weight_sigma.data.fill_(self.sigma_zero / math.sqrt(self.in_features))
      self.bias_mu.data.uniform_(-mu_range, mu_range)
      self.bias_sigma.data.fill_(self.sigma_zero / math.sqrt(self.out_features))
  
    def _scale_noise(self, size):
      x = torch.randn(size)
      return x.sign().mul_(x.abs().sqrt_())
  
    def reset_noise(self):
      epsilon_in = self._scale_noise(self.in_features)
      epsilon_out = self._scale_noise(self.out_features)
      self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
      self.bias_epsilon.copy_(epsilon_out)
  
    def forward(self, input):
      return F.linear(input, self.weight_mu + self.weight_sigma * self.weight_epsilon, self.bias_mu + self.bias_sigma * self.bias_epsilon)
  </code></pre>
  
  <p>However, with the default hyperparameters from the (sigma_0 = 0.5), the agent does not explore at all, and even if I crank it up to sigma_0 = 5, it works way worse than epsilon greedy.<br>
  (When I use noisy nets I don't use epsilon greedy).</p>
  
Tags: <reinforcement-learning><dqn>
Title: NoisyNet DQN with default parameters not exploring
UserDisplayName: Luca Thiede
ThreadText: # NoisyNet DQN with default parameters not exploring
  
  (Asked by: username_0 on 2020-01-14)
  
  I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q value based on the encoded state.  
  My `NoisyLinear` layers looks like this:
  
  ```
  class NoisyLinear(nn.Module):
    def __init__(self, in_features, out_features):
      super(NoisyLinear, self).__init__()
      self.in_features = in_features
      self.out_features = out_features
      self.sigma_zero = 0.5
      self.weight_mu = torch.empty(out_features, in_features)
      self.weight_sigma = torch.empty(out_features, in_features)
      self.weight_epsilon = torch.empty(out_features, in_features, requires_grad=False)
      self.bias_mu = torch.empty(out_features)
      self.bias_sigma = torch.empty(out_features)
      self.bias_epsilon = torch.empty(out_features, requires_grad=False)
      self.reset_parameters()
      self.reset_noise()
  
    def reset_parameters(self):
      mu_range = 1 / math.sqrt(self.in_features)
      self.weight_mu.data.uniform_(-mu_range, mu_range)
      self.weight_sigma.data.fill_(self.sigma_zero / math.sqrt(self.in_features))
      self.bias_mu.data.uniform_(-mu_range, mu_range)
      self.bias_sigma.data.fill_(self.sigma_zero / math.sqrt(self.out_features))
  
    def _scale_noise(self, size):
      x = torch.randn(size)
      return x.sign().mul_(x.abs().sqrt_())
  
    def reset_noise(self):
      epsilon_in = self._scale_noise(self.in_features)
      epsilon_out = self._scale_noise(self.out_features)
      self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
      self.bias_epsilon.copy_(epsilon_out)
  
    def forward(self, input):
      return F.linear(input, self.weight_mu + self.weight_sigma * self.weight_epsilon, self.bias_mu + self.bias_sigma * self.bias_epsilon)
  
  ```
  
  However, with the default hyperparameters from the (sigma\_0 = 0.5), the agent does not explore at all, and even if I crank it up to sigma\_0 = 5, it works way worse than epsilon greedy.  
  (When I use noisy nets I don't use epsilon greedy).
  
  **Tags:** reinforcement-learning, dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-16. Score: 0
  
  I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.
  
  These lines should do the trick:
  
  ```
  self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
  self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
  self.bias_mu = Parameter(torch.Tensor(out_features))
  self.bias_sigma = Parameter(torch.Tensor(out_features))
  
  ```
  
  In case you are not familiar, the `Parameter` class must be imported from `torch`:
  
  ```
  from torch.nn.parameter import Parameter
  
  ```
  
  
  ---
  
  