Answers.Body: <p>That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, <a href="https://arxiv.org/abs/1811.03804" rel="nofollow noreferrer">Gradient Descent Finds Global Minima of Deep Neural Networks</a>, <a href="https://arxiv.org/abs/1811.03962" rel="nofollow noreferrer">A Convergence Theory for Deep Learning via Over-Parameterization</a> or <a href="https://arxiv.org/abs/1811.08888" rel="nofollow noreferrer">Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks</a>). All of the proofs assume that the initial weights are assigned randomly according to a Gaussian distribution. The main reasons this initial distribution is important for the proofs are:</p>
  <ol>
  <li><p>Random weights make the ReLU operators in each layer statistically compressive mapping (up to a linear transformation).</p>
  </li>
  <li><p>Random weights preserve separation of input for any input distribution - that is if input samples are distinguishable network propagation will not make them indistinguishable.</p>
  </li>
  </ol>
  <p>Those properties are very difficult to reproduce with deterministically generated initial weight matrices, and even if they are reproducible with deterministic matrices NULL-space (from which we can generate adversarial examples) would likely make the method less useful in practice. More importantly, preservation of those properties during gradient descent would likely make method impractical. But overall it's very difficult but not impossible, and may warrant some research in that direction. In analogous situation, there were some results for <a href="https://arxiv.org/abs/1202.1234" rel="nofollow noreferrer">Restricted Isometry Property for deterministic matrices</a> in a <a href="https://en.wikipedia.org/wiki/Compressed_sensing" rel="nofollow noreferrer">compressed sensing</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-23T05:52:13.563Z
Answers.LastActivityDate: 2021-01-22T00:58:37.820Z
Answers.LastEditDate: 2021-01-22T00:58:37.820Z
Answers.UserDisplayName: mirror2image
Answers.Body: <p>The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to the same value. While not all of the methods to train neural networks are gradient based, most of them are, and it has been shown in several cases that initializing the neural network to the same value makes the network take much longer to converge on an optimum solution. Also, if you want to retrain your neural network because it got stuck in a local minima, it will get stuck in the same local minima. For the above reasons, we do not set the initial weights to a constant value.</p>
  
  <p>References:
  <a href="https://stats.stackexchange.com/q/45087/82135">Why doesn't backpropagation work when you initialize the weights the same value?</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-23T18:19:34.300Z
Answers.Text: In fact, they break down if all weights are the same.
Answers.UserDisplayName: Quonux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-22T21:14:29.857Z
Answers.LastActivityDate: 2019-04-23T13:49:12.397Z
Answers.LastEditDate: 2019-04-23T13:49:12.397Z
Answers.UserDisplayName: Aiden Grossman
Answers.Body: <p>It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't work.</p>
  <p>Assume here that you have in a 2-layered network with just 2 input features and in the hidden layer you have 2 nodes, and the weights are initialized as [u, v] for both the nodes.</p>
  <p>So, while optimizing the weights, using gradient descent you would compute:</p>
  <p><img src="https://latex.codecogs.com/svg.image?w_%7B1%7D&amp;space;=&amp;space;u&amp;space;-&amp;space;%5Calpha&amp;space;%5Cfrac%7B%5Cpartial&amp;space;J%7D%7B%5Cpartial&amp;space;w_%7B1%7D%7D" alt="" /></p>
  <p><img src="https://latex.codecogs.com/svg.image?w_%7B2%7D&amp;space;=&amp;space;v&amp;space;-&amp;space;%5Calpha&amp;space;%5Cfrac%7B%5Cpartial&amp;space;J%7D%7B%5Cpartial&amp;space;w_%7B2%7D%7D" alt="" /></p>
  <p>where α is the learning rate and J is the cost function that you want to minimize by optimizing the weights.</p>
  <p>By kind of a proof by induction, you should be able to see that after every single iteration of training, your two hidden units are still computing exactly the same function because both hidden units start off computing the same function, have the same influence on the output unit. So even after multiple iterations, the two hidden units are still symmetric. Hence, there's really no point to having more than one hidden unit because they are all computing the same thing.</p>
  <p>In summary, no matter how long you run gradient descent, both the two units compute exactly the same function which is not helpful, because you want the different hidden units to compute different functions. The solution to this is to initialize your parameters randomly.</p>
  <p>For larger neural networks, say of three features and maybe a very large number of hidden units, a similar argument still works.</p>
  <p>Note: It's okay to initialize the bias term to just zeros. Because so long as weights are initialized randomly, you start off with the different hidden units computing different things.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-01T12:26:06.540Z
Answers.LastActivityDate: 2022-07-01T12:34:49.970Z
Answers.LastEditDate: 2022-07-01T12:34:49.970Z
Answers.UserDisplayName: ksgr5566
Answers.Body: <p>Another theory relevant I think to this question is the “lottery ticket hypothesis”:</p>
  <ul>
  <li>Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, 2019, <a href="https://arxiv.org/abs/1803.03635" rel="nofollow noreferrer">https://arxiv.org/abs/1803.03635</a></li>
  </ul>
  <p>Basically, when you have a neural network with a very high number of neurons with randomly initialized weights in the hidden layers, you have a high chance of a subset of the network being close to a good solution to the problem. Then, by applying gradient descent, you can align the output of the whole network to that of the sub-network and fine tune it.</p>
  <p>It would be nice if someone could prove this. If there is a proof I would be interested to see it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T01:57:11.310Z
Answers.LastActivityDate: 2023-03-16T01:57:11.310Z
Answers.UserDisplayName: Bill
Answers.Body: <p>You shouldn't assign all to 0.5 because you'd have the &quot;break symmetry&quot; issue.</p>
  <blockquote>
  <p><a href="http://www.deeplearningbook.org/contents/optimization.html" rel="noreferrer">http://www.deeplearningbook.org/contents/optimization.html</a></p>
  <p>Perhaps the only property known with complete certainty is that the initial
  parameters need to “<strong>break symmetry</strong>” between different units. If two hidden
  units with the same activation function are connected to the same inputs, then
  these units must have <strong>different initial parameters</strong>. If they have the same initial
  parameters, then a deterministic learning algorithm applied to a deterministic cost
  and model will constantly update both of these units in the same way. Even if the
  model or training algorithm is capable of using stochasticity to compute different
  updates for different units (for example, if one trains with dropout), it is usually
  best to initialize each unit to compute a different function from all of the other
  units. This may help to make sure that no input patterns are lost in the null
  space of forward propagation and no gradient patterns are lost in the null space
  of back-propagation.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-24T00:35:12.690Z
Answers.LastActivityDate: 2017-10-24T01:33:39.163Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: SmallChess
Body: <p>This might sound silly to someone who has plenty of experience with neural networks but it bothers me...</p>
  <p>Random initial weights might give you better results that would be somewhat closer to what a trained neural network should look like, but it might as well be the exact opposite of what it should be, while 0.5, or some other average for the range of reasonable weights' values, would sound like a good default setting.</p>
  <p><strong>Why are the initial weights of neural networks randomly initialized rather than being all set to, for example, 0.5?</strong></p>
  
Tags: <neural-networks><training><weights><weights-initialization>
Title: Why are the initial weights of neural networks randomly initialised?
UserDisplayName: Matas Vaitkevicius
ThreadText: # Why are the initial weights of neural networks randomly initialised?
  
  (Asked by: username_0 on 2017-10-21)
  
  This might sound silly to someone who has plenty of experience with neural networks but it bothers me...
  
  Random initial weights might give you better results that would be somewhat closer to what a trained neural network should look like, but it might as well be the exact opposite of what it should be, while 0.5, or some other average for the range of reasonable weights' values, would sound like a good default setting.
  
  **Why are the initial weights of neural networks randomly initialized rather than being all set to, for example, 0.5?**
  
  **Tags:** neural-networks, training, weights, weights-initialization
  
  ---
  
  ## 5 answers
  
  ### Answer by username_6 on 2017-10-24. Score: 13 | Accepted answer
  
  You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue.
  
  > <http://www.deeplearningbook.org/contents/optimization.html>
  >
  > Perhaps the only property known with complete certainty is that the initial
  > parameters need to “**break symmetry**” between different units. If two hidden
  > units with the same activation function are connected to the same inputs, then
  > these units must have **different initial parameters**. If they have the same initial
  > parameters, then a deterministic learning algorithm applied to a deterministic cost
  > and model will constantly update both of these units in the same way. Even if the
  > model or training algorithm is capable of using stochasticity to compute different
  > updates for different units (for example, if one trains with dropout), it is usually
  > best to initialize each unit to compute a different function from all of the other
  > units. This may help to make sure that no input patterns are lost in the null
  > space of forward propagation and no gradient patterns are lost in the null space
  > of back-propagation.
  
  
  ---
  
  ### Answer by username_2 on 2017-10-22. Score: 7
  
  The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to the same value. While not all of the methods to train neural networks are gradient based, most of them are, and it has been shown in several cases that initializing the neural network to the same value makes the network take much longer to converge on an optimum solution. Also, if you want to retrain your neural network because it got stuck in a local minima, it will get stuck in the same local minima. For the above reasons, we do not set the initial weights to a constant value.
  
  References:
  [Why doesn't backpropagation work when you initialize the weights the same value?](https://stats.stackexchange.com/q/45087/82135)
  
  > **Comments:**
  > **username_3**: In fact, they break down if all weights are the same.  
  
  ---
  
  ### Answer by username_1 on 2019-04-23. Score: 5
  
  That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, [Gradient Descent Finds Global Minima of Deep Neural Networks](https://arxiv.org/abs/1811.03804), [A Convergence Theory for Deep Learning via Over-Parameterization](https://arxiv.org/abs/1811.03962) or [Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks](https://arxiv.org/abs/1811.08888)). All of the proofs assume that the initial weights are assigned randomly according to a Gaussian distribution. The main reasons this initial distribution is important for the proofs are:
  
  1. Random weights make the ReLU operators in each layer statistically compressive mapping (up to a linear transformation).
  2. Random weights preserve separation of input for any input distribution - that is if input samples are distinguishable network propagation will not make them indistinguishable.
  
  Those properties are very difficult to reproduce with deterministically generated initial weight matrices, and even if they are reproducible with deterministic matrices NULL-space (from which we can generate adversarial examples) would likely make the method less useful in practice. More importantly, preservation of those properties during gradient descent would likely make method impractical. But overall it's very difficult but not impossible, and may warrant some research in that direction. In analogous situation, there were some results for [Restricted Isometry Property for deterministic matrices](https://arxiv.org/abs/1202.1234) in a [compressed sensing](https://en.wikipedia.org/wiki/Compressed_sensing).
  
  
  ---
  
  ### Answer by username_4 on 2022-07-01. Score: 2
  
  It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't work.
  
  Assume here that you have in a 2-layered network with just 2 input features and in the hidden layer you have 2 nodes, and the weights are initialized as [u, v] for both the nodes.
  
  So, while optimizing the weights, using gradient descent you would compute:
  
  ![](https://latex.codecogs.com/svg.image?w_%7B1%7D&space;=&space;u&space;-&space;%5Calpha&space;%5Cfrac%7B%5Cpartial&space;J%7D%7B%5Cpartial&space;w_%7B1%7D%7D)
  
  ![](https://latex.codecogs.com/svg.image?w_%7B2%7D&space;=&space;v&space;-&space;%5Calpha&space;%5Cfrac%7B%5Cpartial&space;J%7D%7B%5Cpartial&space;w_%7B2%7D%7D)
  
  where α is the learning rate and J is the cost function that you want to minimize by optimizing the weights.
  
  By kind of a proof by induction, you should be able to see that after every single iteration of training, your two hidden units are still computing exactly the same function because both hidden units start off computing the same function, have the same influence on the output unit. So even after multiple iterations, the two hidden units are still symmetric. Hence, there's really no point to having more than one hidden unit because they are all computing the same thing.
  
  In summary, no matter how long you run gradient descent, both the two units compute exactly the same function which is not helpful, because you want the different hidden units to compute different functions. The solution to this is to initialize your parameters randomly.
  
  For larger neural networks, say of three features and maybe a very large number of hidden units, a similar argument still works.
  
  Note: It's okay to initialize the bias term to just zeros. Because so long as weights are initialized randomly, you start off with the different hidden units computing different things.
  
  
  ---
  
  ### Answer by username_5 on 2023-03-16. Score: 1
  
  Another theory relevant I think to this question is the “lottery ticket hypothesis”:
  
  * Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, 2019, <https://arxiv.org/abs/1803.03635>
  
  Basically, when you have a neural network with a very high number of neurons with randomly initialized weights in the hidden layers, you have a high chance of a subset of the network being close to a good solution to the problem. Then, by applying gradient descent, you can align the output of the whole network to that of the sub-network and fine tune it.
  
  It would be nice if someone could prove this. If there is a proof I would be interested to see it.
  
  
  ---
  
  