Answers.Body: <p>Note your case of numerator <span class="math-container">$2$</span> is chosen for normal Xavier initialization, there're other choice such as <span class="math-container">$6$</span> for uniform Xavier initialization. So the heuristic intuition behind the smaller number <span class="math-container">$2$</span> is related to applicable deep feedforward neural network architectures. When initializing weights for activation functions like sigmoid and tanh the normal Xavier initialization technique is suitable because these functions have derivatives that are relatively small compared to the linear activation functions such as relu. For inputs that are far from zero, the derivative of the sigmoid or and tanh function becomes very small or approaching zero to cause vanishing gradient or activation saturation problems, therefore the variance of initialized weights ought to be scaled smaller to keep activations within a reasonable range regardless of the layer sizes.</p>
  <p>In summary the formula here is somewhat heuristic and there might be cases where other methods, or even different forms of the Xavier initialization, work well for specific architectures or datasets.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T15:33:49.537Z
Answers.Text: Could it also be that if one layer is much bigger than the other then that means the input or activation is being looked at in a bunch of different ways in the bigger layer so the variance of the weights decreases rapidly with the formula they used indicating that maybe you don’t need that much variance if you have a ton of neurons because the input or activation is being looked at in a bunch of different ways
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T22:43:13.983Z
Answers.Text: See the Xavier paper: "*For tanh networks, the proposed normalized initialization can be quite helpful, presumably because the layer-to-layer transformations maintain magnitudes of activations (flowing upward) and gradients (flowing backward)... we have used the same number of units per layer. However, we verified that we obtain
  the same gains when the layer size increases (or decreases) with layer number*". Thus your intuition about the relative layer size is incorrect, the desirable thing for DL weight initialization is to maintain variance from above across layers such as Var(z)~=Var(X).
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T22:47:19.280Z
Answers.Text: In Xavier initialization, the key factor is the number of inputs and outputs in the layers and not so much the method of randomization. The goal is to maintain the variance in bounds that enable effective learning with various activation functions.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T23:01:58.660Z
Answers.Text: So in froward prop if x has variance var(X) and W has a variance Var(w)=$1/n_{in}$ when you do forward prop for a layer and then compute the variance for z will it be $n_{in}*Var(x)*1/n_{in}=Var(Z)$ so then by defualt $Var(z)=Var(x)$ or does the variance not directly multiply ?     (I am only considering keeping the variance the same in the froward prop so I can simplify the reasoning I know I need to take into account the backprop variance as well)
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T23:37:23.780Z
Answers.Text: Now you're correct. The ith layerwise variance is derived by equation (5) in the same paper, so to achieve above variance stability goal equation (10) which is same as your expression must hold for each layer. Hope this clarifies and helpful.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T23:46:32.933Z
Answers.Text: and so the equation $2/(n_{in}+n_{out})$ is using the harmonic mean to find a proper balance so that the variance doesn't get explode or diminish on the way forward or backward ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-20T00:07:53.783Z
Answers.Text: Obviouslya your specifically concerned formula is not exactly harmonic mean of nin and nout but resemble harmonic mean, as explained in my answer it's mainly derived empirically to strike a balance between avoiding vanishing gradients (if var(z) is too small) and preventing exploding gradients (if var(z) is too large). Comments are already too long. Please accept and close the question if you're satisfied.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T17:44:14.550Z
Answers.Text: The variance formula derived in the paper is independent of the choice of distribution to initialize the weights from. The 6 in their uniform distribution is just to produce a distribution with variance 2/(n_in + n_out). Additionally, the "compromise" they give _is_ the harmonic mean of 1/n_in and 1/n_out, the variances that would be selected by (10) and (11) respectively. (I am as confused as OP about why they chose this "compromise" in paprticular.)
Answers.UserDisplayName: Paul VanKoughnett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T21:37:50.237Z
Answers.Text: @PaulVanKoughnett Eq (12) happens to be the harmonic mean of 1/nin and 1/nout (not nin and nout). Generally harmonic mean is suitable for rates where both 1/nin and 1/nout can be interpreted as in/out flow rates of a fully connected layer, while arithmetic mean generally more suitable with normally distributed data and is always >= harmonic mean. As I explained in my answer for sigmoid/tanh activations the initial weights should be smaller, that's the only 'intuition' here, other than this, as you see one of the final conclusions of the paper admitted there's still many myths in the scheme.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T00:34:05.080Z
Answers.Text: Yes, that's what I said. The intuition about rates is interesting, though -- is there any justification in thinking of these numbers as flow rates? By the way, I found the discussion in the Kaiming initialization paper a little more understandable than in this paper. It implies that just using variance 1/n_in or 1/n_out shouldn't affect anything for common network designs (in the other direction, the variance will scale by a reasonable constant).
Answers.UserDisplayName: Paul VanKoughnett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T04:47:14.020Z
Answers.Text: @PaulVanKoughnett It's very rough analogy with incompressible fluid dynamics, stable training is like to *keep* the *incompressible* fluid in all the weight pipelines flow forward and backward for their respective passes. Variance of weights are like flow rates, if harmonic average rates are not followed between layers it cannot keep its incompressibility on the 2-way for a long time which is necessary for DL, thus unstable. But these are very speculative together with special physics knowledge and shouldn't further confuse the public here unnecessarily, better left for private discussions.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T06:39:51.777Z
Answers.LastActivityDate: 2024-02-19T06:39:51.777Z
Answers.UserDisplayName: cinch
Answers.Body: <p>More of an extended comment but it felt worth giving as an answer.</p>
  <ol>
  <li>Equations (10) and (11) give you the estimates <span class="math-container">$\mathrm{Var}(W^i) = \frac{1}{n_i}$</span> and <span class="math-container">$\mathrm{Var}(W^i) = \frac{1}{n_{i+1}}$</span> respectively. Equation (12) is the harmonic mean of these estimates.</li>
  <li>As pointed out <a href="https://community.deeplearning.ai/t/question-on-mathematical-intuition-behind-xavier-initialization/573990" rel="nofollow noreferrer">here</a>, one simple way that Glorot and Bengio could have arrived at this &quot;compromise&quot; (without thinking about harmonic means) is to add together equations (10) and (11).</li>
  <li>The paper &quot;Delving Deep into Rectifiers&quot; by He-Zhang-Ren-Sun (<a href="https://ieeexplore.ieee.org/document/7410480" rel="nofollow noreferrer">PDF</a>), which introduces Kaiming initialization, argues that it shouldn't actually matter if one uses <span class="math-container">$\frac{1}{n_i}$</span> or <span class="math-container">$\frac{1}{n_{i+1}}$</span> (or any number between them, like the harmonic mean). Here's the argument: suppose that we pick <span class="math-container">$\mathrm{Var}(W^i) = \frac{1}{n_i}$</span>, so the signal variance is constant through the forward pass. Then over the backward pass, the variance of the gradients scales by a total of <span class="math-container">$$\prod_{i=1}^{L-1} \frac{n_{i+1}}{n_i} = \frac{n_L}{n_1},$$</span> where <span class="math-container">$L$</span> is the number of layers. For most typical neural network designs, this will be within an order of magnitude of 1; also, it won't change if the network gets deeper. So even if the gradients scale somewhat over the backward pass, the scaling isn't <em>exponential</em> and we don't expect vanishing/exploding gradient problems.</li>
  <li>As you hopefully realized, what's in the screenshot is just a calculation of variance, and we still have to choose a distribution. People typically use the uniform or normal distributions with mean 0 and the given variance. The 6 in the numerator of the formula for the &quot;Xavier uniform&quot; distribution is just there to give a distribution with the variance in (12).</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T21:06:50.923Z
Answers.Text: wow that was very helpful cleared up a lot of questions I had
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T22:13:18.250Z
Answers.Text: Xavier initialization scheme with harmonic mean of in/out info flow rates is more conservative than He, even if the gradients variance scaling isn't exponential, in deep layers it may still very likely become large enough to cause sigmoid/tanh activations exploding or vanishing. He scheme is more appropriate for the more common Relu activations since Relu is more stable to become saturated comparatively.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T15:54:35.793Z
Answers.Text: @cinch I think that the important difference between He and Xavier initializations is that the output of a ReLU has roughly half the variance as the output of a sigmoid/tanh, and so He et al. initialize the weights with roughly twice the variance. I would point you to p. 1030 of the He et al. paper (paragraph starting "We note that it is sufficient to use...") for the argument I gave in #3, which applies equally well to non-ReLU activations.
Answers.UserDisplayName: Paul VanKoughnett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-27T20:25:02.310Z
Answers.Text: @PaulVanKoughnett can you please explain the part about how "this will be within an order of maginitude of 1 also it wont change as network get deeper"
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-28T14:48:27.330Z
Answers.Text: @Stef Sure. Say we have a fully-connected network with 5 inputs, a 20-dimensional hidden layer, and 10 outputs. In this case the ratio is 2. So during backpropagation, the variance of the gradients will double -- but this isn't actually a severe enough issue to lead to exploding gradients. Moreover, if we make the network deeper by adding more hidden layers, the ratio doesn't change. You should contrast this to an improperly-initialized network where *every layer* is scaling the gradient variance by 2 -- in this case the gradients would start to explode once the network became deep.
Answers.UserDisplayName: Paul VanKoughnett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-28T17:04:43.930Z
Answers.Text: @PaulVanKoughnett   So basically you can pick either 1/ni or 1/ni+1 and you will have no variance increase on the way forward and then on the way back ward you will have some increase or decrease in variance. But if you don't pick either then variance will explode or vanish both directions a lot ? Is that about right ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-29T19:24:17.700Z
Answers.Text: @Stef Yeah, that's right -- but more generally, if you pick numbers around the same size (e.g. the harmonic mean), then you have controlled variance change on the way forward and back, so you'll still be fine. (Also worth emphasizing that the whole argument only applies to the initial state of the network -- definitely possible for gradients to start vanishing later on in training.)
Answers.UserDisplayName: Paul VanKoughnett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-29T20:21:31.520Z
Answers.Text: Wait I thought it was talking about all training as a whole not just the initial state ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-05T00:31:08.653Z
Answers.Text: @Stef AFAIK there's no guarantee that training will preserve whatever statistical conditions you impose at initialization.
Answers.UserDisplayName: Paul VanKoughnett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T15:10:03.057Z
Answers.LastActivityDate: 2024-02-23T15:10:03.057Z
Answers.UserDisplayName: Paul VanKoughnett
Body: <p>In the research paper on Xavier initialization what is the purpose of putting <span class="math-container">$n_{in}$</span> and <span class="math-container">$n_{out}$</span> under 2 and added them it just says as a compromise but it is not exactly a harmonic mean or an average it seems random ? <a href="https://i.sstatic.net/eIDQm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eIDQm.png" alt="enter image description here" /></a></p>
  
Tags: <machine-learning><weights-initialization>
Title: How Xavier Initialization formula works
UserDisplayName: Stef
ThreadText: # How Xavier Initialization formula works
  
  (Asked by: username_0 on 2024-02-19)
  
  In the research paper on Xavier initialization what is the purpose of putting $n\_{in}$ and $n\_{out}$ under 2 and added them it just says as a compromise but it is not exactly a harmonic mean or an average it seems random ? [![enter image description here](https://i.sstatic.net/eIDQm.png)](https://i.sstatic.net/eIDQm.png)
  
  **Tags:** machine-learning, weights-initialization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-02-23. Score: 2 | Accepted answer
  
  More of an extended comment but it felt worth giving as an answer.
  
  1. Equations (10) and (11) give you the estimates $\mathrm{Var}(W^i) = \frac{1}{n\_i}$ and $\mathrm{Var}(W^i) = \frac{1}{n\_{i+1}}$ respectively. Equation (12) is the harmonic mean of these estimates.
  2. As pointed out [here](https://community.deeplearning.ai/t/question-on-mathematical-intuition-behind-xavier-initialization/573990), one simple way that Glorot and Bengio could have arrived at this "compromise" (without thinking about harmonic means) is to add together equations (10) and (11).
  3. The paper "Delving Deep into Rectifiers" by He-Zhang-Ren-Sun ([PDF](https://ieeexplore.ieee.org/document/7410480)), which introduces Kaiming initialization, argues that it shouldn't actually matter if one uses $\frac{1}{n\_i}$ or $\frac{1}{n\_{i+1}}$ (or any number between them, like the harmonic mean). Here's the argument: suppose that we pick $\mathrm{Var}(W^i) = \frac{1}{n\_i}$, so the signal variance is constant through the forward pass. Then over the backward pass, the variance of the gradients scales by a total of $$\prod\_{i=1}^{L-1} \frac{n\_{i+1}}{n\_i} = \frac{n\_L}{n\_1},$$ where $L$ is the number of layers. For most typical neural network designs, this will be within an order of magnitude of 1; also, it won't change if the network gets deeper. So even if the gradients scale somewhat over the backward pass, the scaling isn't *exponential* and we don't expect vanishing/exploding gradient problems.
  4. As you hopefully realized, what's in the screenshot is just a calculation of variance, and we still have to choose a distribution. People typically use the uniform or normal distributions with mean 0 and the given variance. The 6 in the numerator of the formula for the "Xavier uniform" distribution is just there to give a distribution with the variance in (12).
  
  > **Comments:**
  > **username_0**: wow that was very helpful cleared up a lot of questions I had  
  > **username_1**: Xavier initialization scheme with harmonic mean of in/out info flow rates is more conservative than He, even if the gradients variance scaling isn't exponential, in deep layers it may still very likely become large enough to cause sigmoid/tanh activations exploding or vanishing. He scheme is more appropriate for the more common Relu activations since Relu is more stable to become saturated comparatively.  
  > **username_2**: @username_1 I think that the important difference between He and Xavier initializations is that the output of a ReLU has roughly half the variance as the output of a sigmoid/tanh, and so He et al. initialize the weights with roughly twice the variance. I would point you to p. 1030 of the He et al. paper (paragraph starting "We note that it is sufficient to use...") for the argument I gave in #3, which applies equally well to non-ReLU activations.  
  > **username_0**: @username_2 can you please explain the part about how "this will be within an order of maginitude of 1 also it wont change as network get deeper"  
  > **username_2**: @username_0 Sure. Say we have a fully-connected network with 5 inputs, a 20-dimensional hidden layer, and 10 outputs. In this case the ratio is 2. So during backpropagation, the variance of the gradients will double -- but this isn't actually a severe enough issue to lead to exploding gradients. Moreover, if we make the network deeper by adding more hidden layers, the ratio doesn't change. You should contrast this to an improperly-initialized network where *every layer* is scaling the gradient variance by 2 -- in this case the gradients would start to explode once the network became deep.  
  > **username_0**: @username_2   So basically you can pick either 1/ni or 1/ni+1 and you will have no variance increase on the way forward and then on the way back ward you will have some increase or decrease in variance. But if you don't pick either then variance will explode or vanish both directions a lot ? Is that about right ?  
  > **username_2**: @username_0 Yeah, that's right -- but more generally, if you pick numbers around the same size (e.g. the harmonic mean), then you have controlled variance change on the way forward and back, so you'll still be fine. (Also worth emphasizing that the whole argument only applies to the initial state of the network -- definitely possible for gradients to start vanishing later on in training.)  
  > **username_0**: Wait I thought it was talking about all training as a whole not just the initial state ?  
  > **username_2**: @username_0 AFAIK there's no guarantee that training will preserve whatever statistical conditions you impose at initialization.  
  
  ---
  
  ### Answer by username_1 on 2024-02-19. Score: 1
  
  Note your case of numerator $2$ is chosen for normal Xavier initialization, there're other choice such as $6$ for uniform Xavier initialization. So the heuristic intuition behind the smaller number $2$ is related to applicable deep feedforward neural network architectures. When initializing weights for activation functions like sigmoid and tanh the normal Xavier initialization technique is suitable because these functions have derivatives that are relatively small compared to the linear activation functions such as relu. For inputs that are far from zero, the derivative of the sigmoid or and tanh function becomes very small or approaching zero to cause vanishing gradient or activation saturation problems, therefore the variance of initialized weights ought to be scaled smaller to keep activations within a reasonable range regardless of the layer sizes.
  
  In summary the formula here is somewhat heuristic and there might be cases where other methods, or even different forms of the Xavier initialization, work well for specific architectures or datasets.
  
  > **Comments:**
  > **username_0**: Could it also be that if one layer is much bigger than the other then that means the input or activation is being looked at in a bunch of different ways in the bigger layer so the variance of the weights decreases rapidly with the formula they used indicating that maybe you don’t need that much variance if you have a ton of neurons because the input or activation is being looked at in a bunch of different ways  
  > **username_1**: See the Xavier paper: "*For tanh networks, the proposed normalized initialization can be quite helpful, presumably because the layer-to-layer transformations maintain magnitudes of activations (flowing upward) and gradients (flowing backward)... we have used the same number of units per layer. However, we verified that we obtain the same gains when the layer size increases (or decreases) with layer number*". Thus your intuition about the relative layer size is incorrect, the desirable thing for DL weight initialization is to maintain variance from above across layers such as Var(z)~=Var(X).  
  > **username_1**: In Xavier initialization, the key factor is the number of inputs and outputs in the layers and not so much the method of randomization. The goal is to maintain the variance in bounds that enable effective learning with various activation functions.  
  > **username_0**: So in froward prop if x has variance var(X) and W has a variance Var(w)=$1/n_{in}$ when you do forward prop for a layer and then compute the variance for z will it be $n_{in}*Var(x)*1/n_{in}=Var(Z)$ so then by defualt $Var(z)=Var(x)$ or does the variance not directly multiply ?     (I am only considering keeping the variance the same in the froward prop so I can simplify the reasoning I know I need to take into account the backprop variance as well)  
  > **username_1**: Now you're correct. The ith layerwise variance is derived by equation (5) in the same paper, so to achieve above variance stability goal equation (10) which is same as your expression must hold for each layer. Hope this clarifies and helpful.  
  > **username_0**: and so the equation $2/(n_{in}+n_{out})$ is using the harmonic mean to find a proper balance so that the variance doesn't get explode or diminish on the way forward or backward ?  
  > **username_1**: Obviouslya your specifically concerned formula is not exactly harmonic mean of nin and nout but resemble harmonic mean, as explained in my answer it's mainly derived empirically to strike a balance between avoiding vanishing gradients (if var(z) is too small) and preventing exploding gradients (if var(z) is too large). Comments are already too long. Please accept and close the question if you're satisfied.  
  > **username_2**: The variance formula derived in the paper is independent of the choice of distribution to initialize the weights from. The 6 in their uniform distribution is just to produce a distribution with variance 2/(n_in + n_out). Additionally, the "compromise" they give _is_ the harmonic mean of 1/n_in and 1/n_out, the variances that would be selected by (10) and (11) respectively. (I am as confused as OP about why they chose this "compromise" in paprticular.)  
  > **username_1**: @username_2 Eq (12) happens to be the harmonic mean of 1/nin and 1/nout (not nin and nout). Generally harmonic mean is suitable for rates where both 1/nin and 1/nout can be interpreted as in/out flow rates of a fully connected layer, while arithmetic mean generally more suitable with normally distributed data and is always >= harmonic mean. As I explained in my answer for sigmoid/tanh activations the initial weights should be smaller, that's the only 'intuition' here, other than this, as you see one of the final conclusions of the paper admitted there's still many myths in the scheme.  
  > **username_2**: Yes, that's what I said. The intuition about rates is interesting, though -- is there any justification in thinking of these numbers as flow rates? By the way, I found the discussion in the Kaiming initialization paper a little more understandable than in this paper. It implies that just using variance 1/n_in or 1/n_out shouldn't affect anything for common network designs (in the other direction, the variance will scale by a reasonable constant).  
  > **username_1**: @username_2 It's very rough analogy with incompressible fluid dynamics, stable training is like to *keep* the *incompressible* fluid in all the weight pipelines flow forward and backward for their respective passes. Variance of weights are like flow rates, if harmonic average rates are not followed between layers it cannot keep its incompressibility on the 2-way for a long time which is necessary for DL, thus unstable. But these are very speculative together with special physics knowledge and shouldn't further confuse the public here unnecessarily, better left for private discussions.  
  
  ---
  
  