Answers.Body: <p>I think they use the Xavier/Glorot's initialization method. You can read from <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">the original paper</a>:</p>
  <blockquote>
  <p>We initialized the biases to be 0 and the weights <span class="math-container">$W_{ij}$</span> at each layer with the following commonly used heuristic:</p>
  <p><span class="math-container">$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $</span></p>
  <p>where <span class="math-container">$U[−a, a]$</span> is the uniform distribution in the interval <span class="math-container">$(−a, a)$</span> and <span class="math-container">$n$</span> is the size of the previous layer (the number of columns of <span class="math-container">$W$</span>)</p>
  </blockquote>
  <p>Some people use this as some reports said this initialization method lead to better result</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-10T04:53:38.247Z
Answers.LastActivityDate: 2019-05-10T04:53:38.247Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: malioboro
Body: <p>I have seen in several jupyter notebooks people initializing the NN weights using:</p>
  
  
  
  <pre><code>np.random.randn(D, M) / np.sqrt(D)
  </code></pre>
  
  <p>Other times they just do:</p>
  
  
  
  <pre><code>np.random.randn(D, M)
  </code></pre>
  
  <p>What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?</p>
  
  <p>Thanks</p>
  
Tags: <neural-networks>
Title: Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?
UserDisplayName: Daniel Oliveira
ThreadText: # Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?
  
  (Asked by: username_0 on 2019-05-10)
  
  I have seen in several jupyter notebooks people initializing the NN weights using:
  
  ```
  np.random.randn(D, M) / np.sqrt(D)
  
  ```
  
  Other times they just do:
  
  ```
  np.random.randn(D, M)
  
  ```
  
  What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?
  
  Thanks
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-10. Score: 2 | Accepted answer
  
  I think they use the Xavier/Glorot's initialization method. You can read from [the original paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf):
  
  > We initialized the biases to be 0 and the weights $W\_{ij}$ at each layer with the following commonly used heuristic:
  >
  > $W\_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $
  >
  > where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)
  
  Some people use this as some reports said this initialization method lead to better result
  
  
  ---
  
  