Answers.Body: <p>The most important thing we achieve is indeed making sure the weights are not all equal. If they were, every layer would behave as if it were a single cell.</p>
  <p>We typically want weights that are near zero (so unimportant connections will not accidentally dominate) but non-zero.</p>
  <p>The different types of initialization all have different motivations, including those mentioned in the question.</p>
  <p>If you're curious what the motivation for each one is, I would recommend you check the documentation and try to find the original papers where they were first introduced.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-22T00:15:04.857Z
Answers.Text: So I am getting that one motivation is that unimportant connections will not dominate. I will look into the individual initialization strategies, but what would a perfect weight initialization achieve without having contradictory goals? Is it possible to combine the individual initialization goals?
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-21T23:15:10.173Z
Answers.LastActivityDate: 2020-09-21T23:15:10.173Z
Answers.UserDisplayName: Robby Goetschalckx
Answers.Body: <ul>
  <li><strong>Is it trying to make sure there is no symmetry in the gradients</strong>?</li>
  </ul>
  <p>The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of initialization depending on the dataset type. So, Yes it is trying to avoid symmetry.</p>
  <ul>
  <li><strong>Is it trying to allow the gradients to be large so it can quickly converge?</strong></li>
  </ul>
  <p>The time it takes to converge, is I think a property of the optimizer and not of the weights initialization. Of course, the manner in which we initialize our weights matters but I think Optimization Algorithms contribute more towards convergence</p>
  <ul>
  <li><strong>What are the properties in an initialization that makes the network learn faster?</strong></li>
  </ul>
  <p>Glorot and Bengio believed that Xavier weight initialization would maintain the variance of activations and back-propagated gradients all the way up or down the layers of a network. Incidentally, when they trained deeper networks that used ReLUs, it was found that a 30-layer CNN using Xavier initialization stalled completely and didn’t learn at all. Thus, it depends on the particular problem at hand.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T11:20:50.727Z
Answers.Text: So the goal is somewhat to get every neuron involved in gradient descent? Making sure none of their gradients are very small or zero?
Answers.UserDisplayName: S2673
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T13:09:38.477Z
Answers.Text: Yes, the aim is to have a low number of sparse weights.
Answers.UserDisplayName: Saurav Maheshkar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T03:38:59.967Z
Answers.LastActivityDate: 2020-09-24T03:55:03.840Z
Answers.LastEditDate: 2020-09-24T03:55:03.840Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. <strong>But what is the weight initialization trying to achieve?</strong></p>
  <p>Is it trying to allow the gradients to be large so it can quickly converge? Is it trying to make sure there is no symmetry in the gradients? Is it trying to make the outputs as random as possible to learn more from the loss function? Is it only trying to prevent exploding and vanishing gradients? Is it more about speed or finding a global maximum? What would the perfect weights (without being learned parameters) for a problem achieve? What makes them perfect? What are the properties in an initialization that makes the network learn faster?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-21T23:24:36.707Z
Comments.Text: Finding the "perfect" anything of a black box model is not feasible. You can use a good initialisation method though. As an example, the goal of Xavier initialisation is to maintain a variance at the end of each layer equal to the incoming variance. This helps dramatically with keeping gradients reasonable
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-22T00:10:26.673Z
Comments.Text: @Recessive I’m using perfect very loosely and also I am not asking what the perfect weights would be, but what they would achieve in speeding up learning.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-22T00:17:34.087Z
Comments.Text: It also depends on where the line is drawn. Transfer learning is a form of weight initialisation, and in that case that is an exceptional efficient initialiser. If you only consider random initialisation, I would suggest reading this: https://www.wandb.com/articles/the-effects-of-weight-initialization-on-neural-nets. It's quite comprehensive
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-22T00:21:01.270Z
Comments.Text: @Recessive I actually just read that, but thanks. I am not only considering random initialization. I am not really looking for a specific strategy, but more what a perfect one would be supposed to achieve.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-22T16:58:50.237Z
Comments.Text: This article [Why Initialize a Neural Network with Random Weights?](https://machinelearningmastery.com/why-initialize-a-neural-network-with-random-weights/) should be useful.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><gradient-descent><weights>
Title: What is the goal of weight initialization in neural networks?
UserDisplayName: S2673
ThreadText: # What is the goal of weight initialization in neural networks?
  
  (Asked by: username_0 on 2020-09-21)
  
  This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. **But what is the weight initialization trying to achieve?**
  
  Is it trying to allow the gradients to be large so it can quickly converge? Is it trying to make sure there is no symmetry in the gradients? Is it trying to make the outputs as random as possible to learn more from the loss function? Is it only trying to prevent exploding and vanishing gradients? Is it more about speed or finding a global maximum? What would the perfect weights (without being learned parameters) for a problem achieve? What makes them perfect? What are the properties in an initialization that makes the network learn faster?
  
  **Tags:** neural-networks, machine-learning, gradient-descent, weights
  
  > **Comments:**
  > **username_1**: Finding the "perfect" anything of a black box model is not feasible. You can use a good initialisation method though. As an example, the goal of Xavier initialisation is to maintain a variance at the end of each layer equal to the incoming variance. This helps dramatically with keeping gradients reasonable  
  > **username_0**: @username_1 I’m using perfect very loosely and also I am not asking what the perfect weights would be, but what they would achieve in speeding up learning.  
  > **username_1**: It also depends on where the line is drawn. Transfer learning is a form of weight initialisation, and in that case that is an exceptional efficient initialiser. If you only consider random initialisation, I would suggest reading this: https://www.wandb.com/articles/the-effects-of-weight-initialization-on-neural-nets. It's quite comprehensive  
  > **username_0**: @username_1 I actually just read that, but thanks. I am not only considering random initialization. I am not really looking for a specific strategy, but more what a perfect one would be supposed to achieve.  
  > **username_2**: This article [Why Initialize a Neural Network with Random Weights?](https://machinelearningmastery.com/why-initialize-a-neural-network-with-random-weights/) should be useful.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2020-09-24. Score: 4 | Accepted answer
  
  * **Is it trying to make sure there is no symmetry in the gradients**?
  
  The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of initialization depending on the dataset type. So, Yes it is trying to avoid symmetry.
  
  * **Is it trying to allow the gradients to be large so it can quickly converge?**
  
  The time it takes to converge, is I think a property of the optimizer and not of the weights initialization. Of course, the manner in which we initialize our weights matters but I think Optimization Algorithms contribute more towards convergence
  
  * **What are the properties in an initialization that makes the network learn faster?**
  
  Glorot and Bengio believed that Xavier weight initialization would maintain the variance of activations and back-propagated gradients all the way up or down the layers of a network. Incidentally, when they trained deeper networks that used ReLUs, it was found that a 30-layer CNN using Xavier initialization stalled completely and didn’t learn at all. Thus, it depends on the particular problem at hand.
  
  > **Comments:**
  > **username_0**: So the goal is somewhat to get every neuron involved in gradient descent? Making sure none of their gradients are very small or zero?  
  > **username_4**: Yes, the aim is to have a low number of sparse weights.  
  
  ---
  
  ### Answer by username_3 on 2020-09-21. Score: 1
  
  The most important thing we achieve is indeed making sure the weights are not all equal. If they were, every layer would behave as if it were a single cell.
  
  We typically want weights that are near zero (so unimportant connections will not accidentally dominate) but non-zero.
  
  The different types of initialization all have different motivations, including those mentioned in the question.
  
  If you're curious what the motivation for each one is, I would recommend you check the documentation and try to find the original papers where they were first introduced.
  
  > **Comments:**
  > **username_0**: So I am getting that one motivation is that unimportant connections will not dominate. I will look into the individual initialization strategies, but what would a perfect weight initialization achieve without having contradictory goals? Is it possible to combine the individual initialization goals?  
  
  ---
  
  