Answers.Body: <p><a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">Xavier</a> (or Glorot) initialization is designed to prevent both <code>sigmoid</code> and <code>tanh</code> activations to <em>saturate</em>, i.e., to yield values close to 0 or 1 (for sigmoid) or -1, +1 (for tanh) which causes the gradient during backprop to be zero or so, therefore halting learning. The same happens for the ReLU, when its output is zero. So, <a href="https://openaccess.thecvf.com/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf" rel="nofollow noreferrer">He init</a>, again, tries to prevent that.</p>
  <p>Now, for simple problems and shallow networks you can't really expect big differences, which should be larger for more complex problems (e.g., learn from images or high-dimensional data) and deeper networks, for which He init is also optimized for (think about ResNets that have up to hundreds of layers.)</p>
  <p>In your case, it may have happened that the He init prevented the tanh to saturate (even better than Xavier). But to check that you should plot a per-layer (and per-neuron) histogram of the activations to see that values should <em>not</em> be at the extremes but rather around zero.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-28T13:10:23.073Z
Answers.LastActivityDate: 2024-04-29T11:33:41.633Z
Answers.LastEditDate: 2024-04-29T11:33:41.633Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function.</p>
  <p>In addition, I also learned that Xavier initialization is better than He initialization.</p>
  <p>However, I once implemented a simple MLP with 3 hidden layers using tanh, for function approximation as below:
  2 input neurons
  8 hidden neurons
  1 output neuron</p>
  <p>In the result, He got better result than Xavier, which is not as I expected.</p>
  <p>Their final loss was similar, but He converged faster.</p>
  <p>Can anyone help me why this happens? uniform Xavier and norm Xavier did not showed big difference.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-28T11:21:27.863Z
Comments.Text: the ablation studies on this topic have strong assumptions on the input distribution
Comments.UserDisplayName: Alberto
Tags: <activation-functions><convergence><multilayer-perceptrons><weights-initialization>
Title: Xavier vs He initialization with tanh
UserDisplayName: COTHE
ThreadText: # Xavier vs He initialization with tanh
  
  (Asked by: username_0 on 2024-04-28)
  
  I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function.
  
  In addition, I also learned that Xavier initialization is better than He initialization.
  
  However, I once implemented a simple MLP with 3 hidden layers using tanh, for function approximation as below:
  2 input neurons
  8 hidden neurons
  1 output neuron
  
  In the result, He got better result than Xavier, which is not as I expected.
  
  Their final loss was similar, but He converged faster.
  
  Can anyone help me why this happens? uniform Xavier and norm Xavier did not showed big difference.
  
  **Tags:** activation-functions, convergence, multilayer-perceptrons, weights-initialization
  > **Comments:**
  > **username_1**: the ablation studies on this topic have strong assumptions on the input distribution  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-04-28. Score: 1 | Accepted answer
  
  [Xavier](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) (or Glorot) initialization is designed to prevent both `sigmoid` and `tanh` activations to *saturate*, i.e., to yield values close to 0 or 1 (for sigmoid) or -1, +1 (for tanh) which causes the gradient during backprop to be zero or so, therefore halting learning. The same happens for the ReLU, when its output is zero. So, [He init](https://openaccess.thecvf.com/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf), again, tries to prevent that.
  
  Now, for simple problems and shallow networks you can't really expect big differences, which should be larger for more complex problems (e.g., learn from images or high-dimensional data) and deeper networks, for which He init is also optimized for (think about ResNets that have up to hundreds of layers.)
  
  In your case, it may have happened that the He init prevented the tanh to saturate (even better than Xavier). But to check that you should plot a per-layer (and per-neuron) histogram of the activations to see that values should *not* be at the extremes but rather around zero.
  
  
  ---
  
  