<p>I think they use the Xavier/Glorot's initialization method. You can read from <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">the original paper</a>:</p>
<blockquote>
<p>We initialized the biases to be 0 and the weights <span class="math-container">$W_{ij}$</span> at each layer with the following commonly used heuristic:</p>
<p><span class="math-container">$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $</span></p>
<p>where <span class="math-container">$U[−a, a]$</span> is the uniform distribution in the interval <span class="math-container">$(−a, a)$</span> and <span class="math-container">$n$</span> is the size of the previous layer (the number of columns of <span class="math-container">$W$</span>)</p>
</blockquote>
<p>Some people use this as some reports said this initialization method lead to better result</p>

CC BY-SA 4.0
2019-05-10T04:53:38.247Z
2019-05-10T04:53:38.247Z
2020-06-17T09:57:20.710Z
malioboro
<p>I have seen in several jupyter notebooks people initializing the NN weights using:</p>



<pre><code>np.random.randn(D, M) / np.sqrt(D)
</code></pre>

<p>Other times they just do:</p>



<pre><code>np.random.randn(D, M)
</code></pre>

<p>What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?</p>

<p>Thanks</p>

<neural-networks>
Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?
Daniel Oliveira
# Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?

(Asked by: username_0 on 2019-05-10)

I have seen in several jupyter notebooks people initializing the NN weights using:
np.random.randn(D, M) / np.sqrt(D)

Other times they just do:
np.random.randn(D, M)

What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?
Thanks

**Tags:** neural-networks


---

## 1 answers

### Answer by username_1 on 2019-05-10. Score: 2 | Accepted answer

I think they use the Xavier/Glorot's initialization method. You can read from [the original paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf):

> We initialized the biases to be 0 and the weights $W\_{ij}$ at each layer with the following commonly used heuristic:
>
> $W\_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $
>
> where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result


---

