Answers.Body: <p>Weight initialization is one of the most critical factors for successfully training a deep neural network. This <a href="https://www.deeplearning.ai/ai-notes/initialization/" rel="nofollow noreferrer">explanation</a> by deeplearning.ai is probably the best that one could give for the need for initializing a DNN with Xavier initialization. Here is what it talks about in a nutshell:</p>
  
  <p>The problem of exploding and vanishing gradients has been long-standing in the DL community. Initialize all the weights as zeros and the model learns identical features across all hidden layers, initialize random but large weights and the backpropagated gradients explode, initialize random but small weights and gradients vanish. The intuition is aptly captured by this simple mathematical observation: <span class="math-container">$1.1^{50} = 117.390$</span>, while at the same time, <span class="math-container">$0.9^{50} = 0.00515$</span>. Note that the difference between the two numbers is just <span class="math-container">$0.1$</span> but it has a tremendous effect when multiplied repeatedly! A typical NN is a series of function compositions involving weight matrices and linear/non-linear activation functions. When stripped to a bare minimum, it essentially is a series of matrix multiplications. Therefore, the way in which the elements of these weight matrices are initialized plays a major role in how the network learns.</p>
  
  <p>The standard weight initialization methods come into the picture here. They reinforce what are the de-facto rules of thumb when it comes to weight initializations: (1) the mean of the activations should be zero, and (2) the variance of these activations across all the layers should be the same.</p>
  
  <p>Note: The link given above has complete mathematical justification for why Xavier initialization works, along with an interactive visualization for the same.    </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-13T15:59:24.000Z
Answers.Text: But even with Xavier initialization we can have vanishing gradients. For example in a FC with 10 layers and 1 neuron per layer, each neuron will be initialized from $\mathcal{N} (0, 1)$. These numbers can still be small, and as such their product $0.5^{10}$ can also be small.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T15:45:54.997Z
Answers.LastActivityDate: 2020-05-31T15:45:54.997Z
Answers.UserDisplayName: nagaK
Body: <blockquote>
    <p>The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network</p>
  </blockquote>
  
  <p>I am really having trouble understanding weights initialization technique and Xavier Initialization for deep neural networks (DNNs). </p>
  
  <p>In simple words (and maybe with an example), what is the intuition behind the Xavier initialization for DNNs? When should we use Xavier's initialization?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-30T16:03:04.643Z
Comments.Text: An excellent explanation is available at https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79
Comments.UserDisplayName: Gerry P
Tags: <neural-networks><deep-learning>
Title: What is the intuition behind the Xavier initialization for deep neural networks?
UserDisplayName: Pluviophile
ThreadText: # What is the intuition behind the Xavier initialization for deep neural networks?
  
  (Asked by: username_0 on 2020-05-30)
  
  > The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network
  
  I am really having trouble understanding weights initialization technique and Xavier Initialization for deep neural networks (DNNs).
  
  In simple words (and maybe with an example), what is the intuition behind the Xavier initialization for DNNs? When should we use Xavier's initialization?
  
  **Tags:** neural-networks, deep-learning
  > **Comments:**
  > **username_1**: An excellent explanation is available at https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-31. Score: 1 | Accepted answer
  
  Weight initialization is one of the most critical factors for successfully training a deep neural network. This [explanation](https://www.deeplearning.ai/ai-notes/initialization/) by deeplearning.ai is probably the best that one could give for the need for initializing a DNN with Xavier initialization. Here is what it talks about in a nutshell:
  
  The problem of exploding and vanishing gradients has been long-standing in the DL community. Initialize all the weights as zeros and the model learns identical features across all hidden layers, initialize random but large weights and the backpropagated gradients explode, initialize random but small weights and gradients vanish. The intuition is aptly captured by this simple mathematical observation: $1.1^{50} = 117.390$, while at the same time, $0.9^{50} = 0.00515$. Note that the difference between the two numbers is just $0.1$ but it has a tremendous effect when multiplied repeatedly! A typical NN is a series of function compositions involving weight matrices and linear/non-linear activation functions. When stripped to a bare minimum, it essentially is a series of matrix multiplications. Therefore, the way in which the elements of these weight matrices are initialized plays a major role in how the network learns.
  
  The standard weight initialization methods come into the picture here. They reinforce what are the de-facto rules of thumb when it comes to weight initializations: (1) the mean of the activations should be zero, and (2) the variance of these activations across all the layers should be the same.
  
  Note: The link given above has complete mathematical justification for why Xavier initialization works, along with an interactive visualization for the same.
  
  > **Comments:**
  > **username_3**: But even with Xavier initialization we can have vanishing gradients. For example in a FC with 10 layers and 1 neuron per layer, each neuron will be initialized from $\mathcal{N} (0, 1)$. These numbers can still be small, and as such their product $0.5^{10}$ can also be small.  
  
  ---
  
  