Answers.Body: <p>Weight initialization can and often does matter, hence why pre-training language models are useful for downstream tasks.</p>
  <p>A randomly initialized model is not guaranteed to converge, especially if the model is training on some downstream task with limited data. Furthermore, there is no best practice initialization for weights, but people tend to fall into two camps. Initialize from zero or from some random distribution. However, de-noising objectives can help improve initial weights for downstream tasks.</p>
  <p>Example of benefits of pre-training:
  I pre-trained a model on 97M examples of molecules formatted in a particular molecular language called SMILES, and then trained the model on reaction prediction. Without pre-training, accuracy after just one epoch was .284%; with pre-training, it was 51%. This demonstrates that non-random initialized weights are preferable (in this case), to random ones.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T21:09:36.207Z
Answers.LastActivityDate: 2023-05-18T21:09:36.207Z
Answers.UserDisplayName: Thomas K
Answers.Body: <p><em>Progress about how to best initialize the weights, is what has made neural networks to be popular again.</em></p>
  <p>Initially (around the 80s I think), NNs were initialized from Normal distributions like <span class="math-container">$\mathcal{N}(0, I)$</span>, but that caused unstable training prone to divergence. Also, initializing the weights (excluding the biases) to be <em>constant</em> is something you would NEVER do because the weights with same values are updated in the exactly the same way: so these will be just redundant, not helping the network to learn. Therefore, you want to pick random weights, but how?</p>
  <blockquote>
  <p>Are there weight initialization methods theoretically proven to be worse than others?</p>
  </blockquote>
  <blockquote>
  <p>Is there research into what weight initialization methods work best across the board for different kinds of architectures?</p>
  </blockquote>
  <p>Consider that the choice of which weight initialization scheme to utilize should be made according to what activation function to use in your model, and in some cases is also related to the network architecture. So, weight init and activation are <em>strictly</em> related.</p>
  <ul>
  <li>This <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">paper</a> introduces what is now called the <strong>Glorot</strong> (or Xavier) <strong>initialization</strong>, also motivating why NNs initialized in the classical (thus wrong) way fail. In few words, the Glorot initialization is designed to work well with saturating activations like the <code>sigmoid</code> and <code>tanh</code>, thus preventing them to saturate (which leads to vanishing gradients) early in training.</li>
  <li>This more recent <a href="https://openaccess.thecvf.com/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf" rel="nofollow noreferrer">paper</a>, instead, introduces the <strong>He initialization</strong> strategy that is designed to work with the <em>rectified activations</em> (ReLU, leaky ReLU, etc), thus reducing the chances of dead units. Indeed, this weight init is also demonstrated along the ResNet architectures.</li>
  </ul>
  <p>Indeed, this does not necessarily mean that e.g. using Glorot with ReLU is always bad. But these two papers provide an analysis of the variance of the weights with different initialization strategies, being designed to keep that in control.</p>
  <p>A note about <em>pre-training</em> (also related to transfer learning): strictly speaking this is not a weight initialization strategy. Indeed, you can pick a popular architecture and download the weights pre-trained on ImageNet (for example); that would give a nice initial point, useful to speed-up convergence, down-steams tasks, and even when having few training data. I consider pre-training to be a second step, because the first time you have to pre-train the model yourself and so starting from a proper weight initialization is beneficial also for pre-training alone.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-22T23:49:28.067Z
Answers.Text: For the constant init, see also https://ai.stackexchange.com/q/6789/2444. Btw, I think it's "Xavier" (I suspect this is just a typo).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T17:54:59.430Z
Answers.LastActivityDate: 2023-05-23T10:43:28.007Z
Answers.LastEditDate: 2023-05-23T10:43:28.007Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In my own experience, weight initialization matters for model convergence.</p>
  <ul>
  <li>Theoretically, can different weight initialization methods eventually converge to the same optimal solution? Are their weight initialization methods theoretically proven to be worse than others?</li>
  <li>Is there research into what weight initialization methods work best across the board for different kinds of architectures?</li>
  </ul>
  
Tags: <machine-learning><convergence><weights-initialization>
Title: What is the impact of the initialization of weights in the performance of a neural network in machine learning?
UserDisplayName: Robin van Hoorn
ThreadText: # What is the impact of the initialization of weights in the performance of a neural network in machine learning?
  
  (Asked by: username_0 on 2023-05-18)
  
  In my own experience, weight initialization matters for model convergence.
  
  * Theoretically, can different weight initialization methods eventually converge to the same optimal solution? Are their weight initialization methods theoretically proven to be worse than others?
  * Is there research into what weight initialization methods work best across the board for different kinds of architectures?
  
  **Tags:** machine-learning, convergence, weights-initialization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-05-19. Score: 4 | Accepted answer
  
  *Progress about how to best initialize the weights, is what has made neural networks to be popular again.*
  
  Initially (around the 80s I think), NNs were initialized from Normal distributions like $\mathcal{N}(0, I)$, but that caused unstable training prone to divergence. Also, initializing the weights (excluding the biases) to be *constant* is something you would NEVER do because the weights with same values are updated in the exactly the same way: so these will be just redundant, not helping the network to learn. Therefore, you want to pick random weights, but how?
  
  > Are there weight initialization methods theoretically proven to be worse than others?
  
  > Is there research into what weight initialization methods work best across the board for different kinds of architectures?
  
  Consider that the choice of which weight initialization scheme to utilize should be made according to what activation function to use in your model, and in some cases is also related to the network architecture. So, weight init and activation are *strictly* related.
  
  * This [paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) introduces what is now called the **Glorot** (or Xavier) **initialization**, also motivating why NNs initialized in the classical (thus wrong) way fail. In few words, the Glorot initialization is designed to work well with saturating activations like the `sigmoid` and `tanh`, thus preventing them to saturate (which leads to vanishing gradients) early in training.
  * This more recent [paper](https://openaccess.thecvf.com/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf), instead, introduces the **He initialization** strategy that is designed to work with the *rectified activations* (ReLU, leaky ReLU, etc), thus reducing the chances of dead units. Indeed, this weight init is also demonstrated along the ResNet architectures.
  
  Indeed, this does not necessarily mean that e.g. using Glorot with ReLU is always bad. But these two papers provide an analysis of the variance of the weights with different initialization strategies, being designed to keep that in control.
  
  A note about *pre-training* (also related to transfer learning): strictly speaking this is not a weight initialization strategy. Indeed, you can pick a popular architecture and download the weights pre-trained on ImageNet (for example); that would give a nice initial point, useful to speed-up convergence, down-steams tasks, and even when having few training data. I consider pre-training to be a second step, because the first time you have to pre-train the model yourself and so starting from a proper weight initialization is beneficial also for pre-training alone.
  
  > **Comments:**
  > **username_3**: For the constant init, see also https://ai.stackexchange.com/q/6789/2444. Btw, I think it's "Xavier" (I suspect this is just a typo).  
  
  ---
  
  ### Answer by username_1 on 2023-05-18. Score: 1
  
  Weight initialization can and often does matter, hence why pre-training language models are useful for downstream tasks.
  
  A randomly initialized model is not guaranteed to converge, especially if the model is training on some downstream task with limited data. Furthermore, there is no best practice initialization for weights, but people tend to fall into two camps. Initialize from zero or from some random distribution. However, de-noising objectives can help improve initial weights for downstream tasks.
  
  Example of benefits of pre-training:
  I pre-trained a model on 97M examples of molecules formatted in a particular molecular language called SMILES, and then trained the model on reaction prediction. Without pre-training, accuracy after just one epoch was .284%; with pre-training, it was 51%. This demonstrates that non-random initialized weights are preferable (in this case), to random ones.
  
  
  ---
  
  