Answers.Body: <p>There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">Xavier et al.</a>. Moreover, there are several numerical experiments showing the importance of initialization.</p>
  
  <p>The motivation for Xavier initialization in Neural Networks is to initialize the weights of the network so that the neuron activation functions are not starting out in saturated or dead regions. In other words, we want to initialize the weights with random values that are not "too small" and not "too large". Thus the purpose is to fix the variance of the input data to each neuron to 1, because this reduces the variance of getting stuck in saturated areas, since the data is in general normalized this is equivalent to fixing the variance of each weight to <span class="math-container">$1/n$</span>, where n is the number of input weights to the given neuron. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-08T10:47:45.960Z
Answers.LastActivityDate: 2020-02-08T11:15:29.277Z
Answers.LastEditDate: 2020-02-08T11:15:29.277Z
Answers.UserDisplayName: hola
Body: <p>How do newer weight initialization techniques (He, Xavier, etc) improve results over zero or random initialization of weights in a neural network? Is there any mathematical evidence behind this?</p>
  
Tags: <neural-networks><deep-learning><weights><weights-initialization>
Title: How are newer weight initialization techniques better than zero or random initialization?
UserDisplayName: jaeger6
ThreadText: # How are newer weight initialization techniques better than zero or random initialization?
  
  (Asked by: username_0 on 2020-02-08)
  
  How do newer weight initialization techniques (He, Xavier, etc) improve results over zero or random initialization of weights in a neural network? Is there any mathematical evidence behind this?
  
  **Tags:** neural-networks, deep-learning, weights, weights-initialization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-08. Score: 3
  
  There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, [Xavier et al.](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf). Moreover, there are several numerical experiments showing the importance of initialization.
  
  The motivation for Xavier initialization in Neural Networks is to initialize the weights of the network so that the neuron activation functions are not starting out in saturated or dead regions. In other words, we want to initialize the weights with random values that are not "too small" and not "too large". Thus the purpose is to fix the variance of the input data to each neuron to 1, because this reduces the variance of getting stuck in saturated areas, since the data is in general normalized this is equivalent to fixing the variance of each weight to $1/n$, where n is the number of input weights to the given neuron.
  
  
  ---
  
  