Answers.Body: <blockquote>
  <p>Are there any theoretical, computational or empirical justifications for when to use a normal over a uniform, or a uniform over a normal weight initialization regardless of the final weight variance?</p>
  </blockquote>
  <p>There is no hard evidence that one initialization works better than the other in certain cases. It seems to be down to experience and empirical results. As you mentioned, there are no computational differences in the initialization itself, and even if there were, it would be minuscule compared to the training itself.</p>
  <p>Even though I typically don't expect the initialization to impact the end results dramatically, here is my rule of thumb when picking uniform vs normal:</p>
  <ul>
  <li>Uniform: when the architecture is small/compact, I expect all weights to &quot;contribute&quot;, so typically smaller architectures.</li>
  </ul>
  <p>The reasoning behind this is that let's say the &quot;optimal&quot; model has all of its weights being non-zero (i.e. all weights are important), when you uniformly initialize, the &quot;distance&quot; between the initial weight and the &quot;ideal&quot; weight will be somewhat shorter and you'd hope to converge quicker.</p>
  <ul>
  <li>Normal: when my architecture is likely &quot;bigger than needed, &quot; typically larger architectures.</li>
  </ul>
  <p>Similarly, if I expect my &quot;ideal&quot; model to have relatively sparse weights (i.e. a lot of them are zeros), I'd like my weights to be as close as possible to that.</p>
  <p>So, in short, your initialization might impact how quickly you converge and, therefore, &quot;where&quot; you would converge. How &quot;sparse&quot; your converged network does impact the model's complexity and robustness, but if you do have a really sparse network at the end, then perhaps your architecture could have been made much smaller in the first place.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-01T15:30:09.967Z
Answers.Text: Thanks for your answer to my 3 year old question. Very appreciated! Can you elaborate what you mean by "distance between initial and ideal weights is shorter"? If I follow your reasoning, I would use uniform for large / sparse models instead of normal. On average both schemes yield 0 weights with same variance, bigger networks may be more sparse after convergence, using normal could produce some high sigma weights at initialization compared to uniform where the weights are bounded. Therefore some weights would be further away from their optimum.
Answers.UserDisplayName: Tinu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-01T16:28:21.233Z
Answers.Text: When I'm talking about distance, it is very much to build up intuition. Let's say that you put all your network's weights into one vector. In some extreme cases, it makes sense that your initialization could be bad (i.e. all your weights are very large) since we have some reasonable expectations about the "ideal" weights. Also, most networks will converge to some local, not global, minimum. So, you're essentially hoping that your weights will be "close" (i.e. using euclidian distance) to a "good" set of weights. Because then, training will be quicker.
Answers.UserDisplayName: Valentin Calomme
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-30T12:03:59.227Z
Answers.LastActivityDate: 2023-12-30T12:03:59.227Z
Answers.UserDisplayName: Valentin Calomme
Body: <p>Two of the most popular initialization schemes for neural network weights today are <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="noreferrer">Xavier</a> and <a href="https://arxiv.org/pdf/1502.01852v1.pdf" rel="noreferrer">He</a>. Both methods propose random weight initialization with a variance dependent on the number of input and output units. Xavier proposes</p>
  <p><span class="math-container">$$W \sim \mathcal{U}\Bigg[-\frac{\sqrt{6}}{\sqrt{n_{in}+n_{out}}},\frac{\sqrt{6}}{\sqrt{n_{in}+n_{out}}}\Bigg]$$</span></p>
  <p>for networks with <span class="math-container">$\text{tanh}$</span> activation function and He proposes</p>
  <p><span class="math-container">$$W \sim \mathcal{N}(0,\sqrt{s/n_{in}})$$</span></p>
  <p>for <span class="math-container">$\text{ReLU}$</span> activation. Both initialization schemes are implemented in the most commonly used deep learning libraries for python, <a href="https://pytorch.org/docs/stable/nn.init.html" rel="noreferrer">PyTorch</a> and <a href="https://www.tensorflow.org/api_docs/python/tf/keras/initializers" rel="noreferrer">TensorFlow</a>.<br />
  However, <strong>for both versions we have a normal and uniform version</strong>. Now the main argument of both papers is about the variance of the information at initialization time (which is dependent on the non-linearity) and that it should stay constant across all layers when back-propagating. I see how one can simply adjust the bounds <span class="math-container">$[-a,a]$</span> of a uniform variable in such a way that the random variable has the desired standard deviation and vice versa (<span class="math-container">$\sigma = a/\sqrt{3}$</span>), but <strong>I'm not sure why we need a normal and a uniform version for both schemes?</strong> Wouldn't it be just enough to have only normal or only uniform? Or uniform Xavier and normal He as proposed in their papers?</p>
  <p>I can imagine uniform distributions are easier to sample from a computational point of view, but since we do the initialization operation only once at the beginning, the computational cost is negligible compared to that from training. Further uniform variables are bounded, so there are no long tail observations as one would expect in a normal. I suppose that's why both libraries have truncated normal initializations.</p>
  <p><strong>Are there any theoretical, computational or empirical justifications for when to use a normal over a uniform, or a uniform over a normal weight initialization regardless of the final weight variance?</strong></p>
  
Tags: <neural-networks><training><weights><weights-initialization>
Title: Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?
UserDisplayName: Tinu
ThreadText: # Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?
  
  (Asked by: username_0 on 2020-12-17)
  
  Two of the most popular initialization schemes for neural network weights today are [Xavier](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) and [He](https://arxiv.org/pdf/1502.01852v1.pdf). Both methods propose random weight initialization with a variance dependent on the number of input and output units. Xavier proposes
  
  $$W \sim \mathcal{U}\Bigg[-\frac{\sqrt{6}}{\sqrt{n\_{in}+n\_{out}}},\frac{\sqrt{6}}{\sqrt{n\_{in}+n\_{out}}}\Bigg]$$
  
  for networks with $\text{tanh}$ activation function and He proposes
  
  $$W \sim \mathcal{N}(0,\sqrt{s/n\_{in}})$$
  
  for $\text{ReLU}$ activation. Both initialization schemes are implemented in the most commonly used deep learning libraries for python, [PyTorch](https://pytorch.org/docs/stable/nn.init.html) and [TensorFlow](https://www.tensorflow.org/api_docs/python/tf/keras/initializers).  
  However, **for both versions we have a normal and uniform version**. Now the main argument of both papers is about the variance of the information at initialization time (which is dependent on the non-linearity) and that it should stay constant across all layers when back-propagating. I see how one can simply adjust the bounds $[-a,a]$ of a uniform variable in such a way that the random variable has the desired standard deviation and vice versa ($\sigma = a/\sqrt{3}$), but **I'm not sure why we need a normal and a uniform version for both schemes?** Wouldn't it be just enough to have only normal or only uniform? Or uniform Xavier and normal He as proposed in their papers?
  
  I can imagine uniform distributions are easier to sample from a computational point of view, but since we do the initialization operation only once at the beginning, the computational cost is negligible compared to that from training. Further uniform variables are bounded, so there are no long tail observations as one would expect in a normal. I suppose that's why both libraries have truncated normal initializations.
  
  **Are there any theoretical, computational or empirical justifications for when to use a normal over a uniform, or a uniform over a normal weight initialization regardless of the final weight variance?**
  
  **Tags:** neural-networks, training, weights, weights-initialization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-30. Score: 2
  
  > Are there any theoretical, computational or empirical justifications for when to use a normal over a uniform, or a uniform over a normal weight initialization regardless of the final weight variance?
  
  There is no hard evidence that one initialization works better than the other in certain cases. It seems to be down to experience and empirical results. As you mentioned, there are no computational differences in the initialization itself, and even if there were, it would be minuscule compared to the training itself.
  
  Even though I typically don't expect the initialization to impact the end results dramatically, here is my rule of thumb when picking uniform vs normal:
  
  * Uniform: when the architecture is small/compact, I expect all weights to "contribute", so typically smaller architectures.
  
  The reasoning behind this is that let's say the "optimal" model has all of its weights being non-zero (i.e. all weights are important), when you uniformly initialize, the "distance" between the initial weight and the "ideal" weight will be somewhat shorter and you'd hope to converge quicker.
  
  * Normal: when my architecture is likely "bigger than needed, " typically larger architectures.
  
  Similarly, if I expect my "ideal" model to have relatively sparse weights (i.e. a lot of them are zeros), I'd like my weights to be as close as possible to that.
  
  So, in short, your initialization might impact how quickly you converge and, therefore, "where" you would converge. How "sparse" your converged network does impact the model's complexity and robustness, but if you do have a really sparse network at the end, then perhaps your architecture could have been made much smaller in the first place.
  
  > **Comments:**
  > **username_0**: Thanks for your answer to my 3 year old question. Very appreciated! Can you elaborate what you mean by "distance between initial and ideal weights is shorter"? If I follow your reasoning, I would use uniform for large / sparse models instead of normal. On average both schemes yield 0 weights with same variance, bigger networks may be more sparse after convergence, using normal could produce some high sigma weights at initialization compared to uniform where the weights are bounded. Therefore some weights would be further away from their optimum.  
  > **username_1**: When I'm talking about distance, it is very much to build up intuition. Let's say that you put all your network's weights into one vector. In some extreme cases, it makes sense that your initialization could be bad (i.e. all your weights are very large) since we have some reasonable expectations about the "ideal" weights. Also, most networks will converge to some local, not global, minimum. So, you're essentially hoping that your weights will be "close" (i.e. using euclidian distance) to a "good" set of weights. Because then, training will be quicker.  
  
  ---
  
  