Answers.Body: <p>I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape.</p>
  <p>Batch and layer normalization would help ensure that the feature vectors (i.e. channels) are embedded around the unit sphere <a href="https://ai.stackexchange.com/questions/35072/batch-instance-norm-translates-to-origin-layer-norm-scales-radially-to-unit-sph">Batch/Instance norm translates to origin. Layer norm scales radially to unit sphere</a>. Viewing neural networks as <a href="https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/" rel="nofollow noreferrer">transformations</a>, this would make the loss landscape smoother since the transformations the neural net needs to find would be more &quot;regular&quot;.</p>
  <p>I would recomend this <a href="https://www.youtube.com/watch?v=78vq6kgsTa8" rel="nofollow noreferrer">video</a> to learn about loss landscapes.</p>
  <p>From <a href="https://arxiv.org/pdf/1712.09913.pdf" rel="nofollow noreferrer">Visualizing the Loss Landscape of Neural Nets. NeuRIPS 2018:</a>
  <img src="https://vitalab.github.io/article/images/lossLandscape/sc01.jpg" alt="" />
  <img src="https://shaojiejiang.github.io/post/en/visualizing-loss/featured_hu77603ba4005b2625759145c28ebb9f68_402754_720x0_resize_lanczos_3.png" alt="" /></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-03T05:48:13.963Z
Answers.LastActivityDate: 2022-04-03T05:55:11.040Z
Answers.LastEditDate: 2022-04-03T05:55:11.040Z
Answers.UserDisplayName: Tom Huntington
Answers.Body: <p>This got me thinking about my understanding of batch normalization. I thought I understand it until I read this. Then, I refer to the Coursera deep learning specialization by Andrew Ng. <br>
  Prof. Andrew Ng explained it this way.</p>
  <hr />
  <p>One reason why does batch norm work is that it normalizes not only the input features but also further values in the hidden units to take
  on a similar range of values that can speed up learning.</p>
  <p>The second reason why <strong>batch norm</strong> works, is it makes weights, later or
  deeper than the network you have, say the weight on layer 10, more robust to changes to weights in earlier layers of the neural network (eg. in layer one). However, these hidden unit values are changing all the time, and so it's suffering from the <strong>problem of covariate shift</strong>.  So <strong>what batch norm does</strong>, is it reduces the amount that the distribution of these hidden unit values shifts around. What batch norm ensures is that no matter how the parameters of the neural network update, their mean and variance will at least stay the same mean and variance, causing the input values to become more stable, so that the later layers of the neural network has more firm ground to stand on.
  <br></p>
  <p>And even though the input distribution changes a bit, it changes less, and
  what this does is, even as the earlier layers keep learning, the amounts that this forces the later layers to adapt
  to as early as layer changes is reduced or, if you will, it weakens the coupling between what the early layers
  parameters has to do and what the later layers parameters have to do. <strong>And so it allows each layer of the
  network to learn by itself, a little bit more independently of other layers, and this has the effect of
  speeding up of learning in the whole network.</strong> Takeaway is that batch norm means that, especially from
  the perspective of one of the later layers of the neural network, the earlier layers don't get to shift around as
  much, because they're constrained to have the same mean and variance. And so this makes the job of learning
  on the later layers easier. It turns out batch norm has a second effect, it has a slight regularization effect. So
  one non-intuitive thing of a batch norm is that each mini-batch, the mean and variance computed on just that
  mini-batch as opposed to computed on the entire data set, that mean and variance has a little bit of noise
  in it, because it's computed just on your mini-batch of, say, 64, or 128, or maybe 256 or larger training
  examples. Batch norm works with mini-batch</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-08T14:58:32.377Z
Answers.Text: The punctuation needs some work.
Answers.UserDisplayName: HappyFace
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-09T04:47:50.717Z
Answers.Text: @HappyFace, I see your point.
Answers.UserDisplayName: kibromhft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-09T10:03:39.653Z
Answers.Text: Hey, @kibromhft, my question specifically mentions these kinds of explanations. Experiments in the 2018 paper show that BatchNorm doesn't affect distributions in inner layers. Even more - **artificially** changing these distributions doesn't affect the performance much.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-08T09:02:56.253Z
Answers.LastActivityDate: 2022-02-08T09:02:56.253Z
Answers.UserDisplayName: kibromhft
Answers.Body: <blockquote>
  <p>When we are training deep neural Network gradient tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously. When we update, unexpected results can happen because many functions composed together are changed simultaneously using updates that were computed under the assumption that other function remains constant.This makes it very hard to choose an appropriate leaning rate, because the effects of an update to the parametrs of one layer strongly on all other layers.</p>
  </blockquote>
  <p><em><strong>How does Batch Normalisation Help :</strong></em>
  Batch Normalisation a layer which is added to any input or hidden layer in the neural network. Suppose H is the minitach of activations of the layer to normalize.</p>
  <p>The formula for normalizing H is :</p>
  <p><span class="math-container">$_H = \frac{H - Mean}{Standard Deviation}$</span></p>
  <p>Mean : Vector Containing Mean of each unit
  Standard Deviation : Vector Containing Mean of each unit</p>
  <p>At training time mean and sd are calculated and when we backpropogate through these operations for apply mean, sd and Normalize H. This means that gradient will never propose an operation that acts simply to increase the standard deviation and mean of hi, the normalization operation remove the effect of such an action and zero out its componenr in the gradient. Hence Batch Normalisation thus ensure no or slight covariance shift in the input to layer after Batch Normalisation and thus improving learning time as shown in the original paper mentioned in question.</p>
  <p>For more details : <a href="https://www.deeplearningbook.org/contents/optimization.html" rel="nofollow noreferrer">https://www.deeplearningbook.org/contents/optimization.html</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T11:39:21.883Z
Answers.Text: At least this "tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously." seems to be copied from [here](https://ai.stackexchange.com/q/20308/2444). Either edit this answer to clarify which parts you copied or this answer will be deleted.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T11:44:36.107Z
Answers.Text: if you look at the reference we are referring to same reference. Nothing has been copied from the link :) The sentence you have highlighted is taken from reference.
Answers.UserDisplayName: Ashwiniku918
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T11:46:14.873Z
Answers.Text: Yes, I understood that the reference is the same, but **you cannot just copy from a reference without explicitly showing us WHAT you copied**, i.e. you need to quote. To quote, you need to precede the text that you copied with `>`. Take a look at https://ai.stackexchange.com/help/referencing to know more about how to quote parts of other answers. Please, take the time to go through all your answers to quote things that you copied (from other sources) in order for them not to be deleted.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T11:50:35.350Z
Answers.Text: Thanks i am new to this platform , will keep this things in mind
Answers.UserDisplayName: Ashwiniku918
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T11:53:41.270Z
Answers.Text: Don't just put the reference at the end. Whenever you quote something, tell us immediately where you copied it from, which section and page, so that people can check that you're referencing a specific part of another source. That chapter 8 is a bit long.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T10:37:01.830Z
Answers.LastActivityDate: 2022-01-06T11:52:12.797Z
Answers.LastEditDate: 2022-01-06T11:52:12.797Z
Answers.UserDisplayName: Ashwiniku918
Answers.Body: <p>It is a question with no simple answer.</p>
  <p>On one hand the BatchNormalization is unloved by some arguing it doesn't change the accuracy of neural networks or biased them.
  On the other hand, it is highly recommended by the other because it leads to better trained models with a larger scope of predictions and less chances of overflow.</p>
  <p>All I know for sure is that BN is really efficient on image classification. In fact, like the image categorization and classification soar this last years and that BN is a good practice in this field, it has spread to almost all DNNs.</p>
  <p>Not only is the BN not always used in the right purpose, but it is often used without taking into account several elements such as :</p>
  <ul>
  <li>The layers between which apply BN</li>
  <li>The initializer algorithms</li>
  <li>The activation algorithms</li>
  <li>etc</li>
  </ul>
  <p>For more computer sciences litterature &quot;against&quot; BN, I will let you look at the <a href="https://arxiv.org/abs/1901.09321" rel="nofollow noreferrer">H. Zhang et al paper</a> who has trained a DNN without BN and get good results.</p>
  <p>Some people use Gradient Clipping technique (R. Pascanu) instead of the BN in particular for RNNs</p>
  <p>I hope it will give you some answers !</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-28T21:29:36.267Z
Answers.Text: Thanks for the response. I'm following it as I added the bounty. I down voted it but don't take offence! The reason I did it is because I don't believe any part of it really addresses the question. Votes help others quickly identify the most relevant responses.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-29T16:40:25.720Z
Answers.Text: Thanks for the feedback and the explanation. So I guess I didn't understand the question. Can you just reformulate it to let me understand it please ?
Answers.UserDisplayName: himiro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-30T08:34:02.187Z
Answers.Text: I think your answer is more for a question like "Where is BN useful vs not?" or "Is BN always a good thing to use?". The question asks "What is the generally accepted reason for why BN works well?" Like if I ask "Why do conv kernels work well for images?" the answer is along the lines of: "Because it satisfies the inductive priors of translational invariance, feature hierarchies, and to some degree, scale invariance". Your answer doesn't talk about the properties of BN that make it successful. And in this case OP also wants the answer to be backed by authoritative sources
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-28T14:00:46.430Z
Answers.LastActivityDate: 2021-06-28T14:00:46.430Z
Answers.UserDisplayName: himiro
Answers.Body: <p>To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-16T18:35:27.147Z
Answers.Text: Adding some _artificial_ noise to neural networks quite often increases their generalization performance.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T14:38:53.387Z
Answers.Text: I was just thinking about the conditioning that BatchNorm performs and that also in some cases this process cutoff low intensity component. When a feature appears on a layer, it become evident since its intensity is very over average. When you "normalize",  all others components falls under threshold at least numerically. Then this "kind of noise" is removed. Another thing is adding *uniform noise*, which helps to search such a desirable feature and it is not removed by "normalization".
Answers.UserDisplayName: Pablo Roman Asenjo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-16T18:16:40.913Z
Answers.LastActivityDate: 2021-04-16T18:16:40.913Z
Answers.UserDisplayName: Pablo Roman Asenjo
Body: <p>Adding <a href="https://en.wikipedia.org/wiki/Batch_normalization" rel="noreferrer">BatchNorm</a> layers improves training time and makes the whole deep model more stable.  That's an experimental fact that is widely used in machine learning practice.</p>
  <p>My question is - why does it work?</p>
  <p>The <a href="https://arxiv.org/abs/1502.03167" rel="noreferrer">original (2015) paper</a> motivated the introduction of the layers by stating that these layers help fixing &quot;<em>internal covariate shift</em>&quot;. The rough idea is that large shifts in the distributions of inputs of inner layers makes training less stable, leading to a decrease in the learning rate and slowing down of the training.  Batch normalization mitigates this problem by standardizing the inputs of inner layers.</p>
  <p>This explanation was harshly criticized <a href="https://arxiv.org/abs/1805.11604" rel="noreferrer">by the next (2018) paper</a> -- quoting the abstract:</p>
  <blockquote>
  <p>... distributional stability of layer inputs has little to do with the success of BatchNorm</p>
  </blockquote>
  <p>They demonstrate that BatchNorm only slightly affects the inner layer inputs distributions.  More than that -- they tried to <em>inject</em> some non-zero mean/variance noise into the distributions. And they still got almost the same performance.</p>
  <p>Their conclusion was that the real reason BatchNorm works was that...</p>
  <blockquote>
  <p>Instead BatchNorm makes the optimization landscape significantly smoother.</p>
  </blockquote>
  <p>Which, to my taste, is slightly tautological to saying that it improves stability.</p>
  <p>I've found two more papers trying to tackle the question: In <a href="https://arxiv.org/abs/2002.10444" rel="noreferrer">this paper</a> the &quot;key benefit&quot; is claimed to be the fact that Batch Normalization biases residual blocks towards the identity function. And in <a href="https://arxiv.org/abs/2003.01652" rel="noreferrer">this paper</a> that it &quot;avoids rank collapse&quot;.</p>
  <p>So, is there any bottom line? Why does BatchNorm work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-12T11:43:20.630Z
Comments.Text: [This](https://blog.paperspace.com/busting-the-myths-about-batch-normalization/) blog post attempted to explain why this is the case (go to the part "So, why does Batch Norm Work?"), but as they mentioned it is kind of a speculative thought.
Comments.UserDisplayName: Enes
Tags: <neural-networks><machine-learning><papers><hidden-layers><batch-normalization>
Title: Why does Batch Normalization work?
UserDisplayName: Kostya
ThreadText: # Why does Batch Normalization work?
  
  (Asked by: username_0 on 2021-04-10)
  
  Adding [BatchNorm](https://en.wikipedia.org/wiki/Batch_normalization) layers improves training time and makes the whole deep model more stable. That's an experimental fact that is widely used in machine learning practice.
  
  My question is - why does it work?
  
  The [original (2015) paper](https://arxiv.org/abs/1502.03167) motivated the introduction of the layers by stating that these layers help fixing "*internal covariate shift*". The rough idea is that large shifts in the distributions of inputs of inner layers makes training less stable, leading to a decrease in the learning rate and slowing down of the training. Batch normalization mitigates this problem by standardizing the inputs of inner layers.
  
  This explanation was harshly criticized [by the next (2018) paper](https://arxiv.org/abs/1805.11604) -- quoting the abstract:
  
  > ... distributional stability of layer inputs has little to do with the success of BatchNorm
  
  They demonstrate that BatchNorm only slightly affects the inner layer inputs distributions. More than that -- they tried to *inject* some non-zero mean/variance noise into the distributions. And they still got almost the same performance.
  
  Their conclusion was that the real reason BatchNorm works was that...
  
  > Instead BatchNorm makes the optimization landscape significantly smoother.
  
  Which, to my taste, is slightly tautological to saying that it improves stability.
  
  I've found two more papers trying to tackle the question: In [this paper](https://arxiv.org/abs/2002.10444) the "key benefit" is claimed to be the fact that Batch Normalization biases residual blocks towards the identity function. And in [this paper](https://arxiv.org/abs/2003.01652) that it "avoids rank collapse".
  
  So, is there any bottom line? Why does BatchNorm work?
  
  **Tags:** neural-networks, machine-learning, papers, hidden-layers, batch-normalization
  
  > **Comments:**
  > **username_1**: [This](https://blog.paperspace.com/busting-the-myths-about-batch-normalization/) blog post attempted to explain why this is the case (go to the part "So, why does Batch Norm Work?"), but as they mentioned it is kind of a speculative thought.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_2 on 2022-04-03. Score: 4
  
  I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape.
  
  Batch and layer normalization would help ensure that the feature vectors (i.e. channels) are embedded around the unit sphere [Batch/Instance norm translates to origin. Layer norm scales radially to unit sphere](https://ai.stackexchange.com/questions/35072/batch-instance-norm-translates-to-origin-layer-norm-scales-radially-to-unit-sph). Viewing neural networks as [transformations](https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/), this would make the loss landscape smoother since the transformations the neural net needs to find would be more "regular".
  
  I would recomend this [video](https://www.youtube.com/watch?v=78vq6kgsTa8) to learn about loss landscapes.
  
  From [Visualizing the Loss Landscape of Neural Nets. NeuRIPS 2018:](https://arxiv.org/pdf/1712.09913.pdf)
  ![](https://vitalab.github.io/article/images/lossLandscape/sc01.jpg)
  ![](https://shaojiejiang.github.io/post/en/visualizing-loss/featured_hu77603ba4005b2625759145c28ebb9f68_402754_720x0_resize_lanczos_3.png)
  
  
  ---
  
  ### Answer by username_3 on 2022-02-08. Score: 0
  
  This got me thinking about my understanding of batch normalization. I thought I understand it until I read this. Then, I refer to the Coursera deep learning specialization by Andrew Ng.   
  Prof. Andrew Ng explained it this way.
  
  ---
  
  One reason why does batch norm work is that it normalizes not only the input features but also further values in the hidden units to take
  on a similar range of values that can speed up learning.
  
  The second reason why **batch norm** works, is it makes weights, later or
  deeper than the network you have, say the weight on layer 10, more robust to changes to weights in earlier layers of the neural network (eg. in layer one). However, these hidden unit values are changing all the time, and so it's suffering from the **problem of covariate shift**. So **what batch norm does**, is it reduces the amount that the distribution of these hidden unit values shifts around. What batch norm ensures is that no matter how the parameters of the neural network update, their mean and variance will at least stay the same mean and variance, causing the input values to become more stable, so that the later layers of the neural network has more firm ground to stand on.
  
  And even though the input distribution changes a bit, it changes less, and
  what this does is, even as the earlier layers keep learning, the amounts that this forces the later layers to adapt
  to as early as layer changes is reduced or, if you will, it weakens the coupling between what the early layers
  parameters has to do and what the later layers parameters have to do. **And so it allows each layer of the
  network to learn by itself, a little bit more independently of other layers, and this has the effect of
  speeding up of learning in the whole network.** Takeaway is that batch norm means that, especially from
  the perspective of one of the later layers of the neural network, the earlier layers don't get to shift around as
  much, because they're constrained to have the same mean and variance. And so this makes the job of learning
  on the later layers easier. It turns out batch norm has a second effect, it has a slight regularization effect. So
  one non-intuitive thing of a batch norm is that each mini-batch, the mean and variance computed on just that
  mini-batch as opposed to computed on the entire data set, that mean and variance has a little bit of noise
  in it, because it's computed just on your mini-batch of, say, 64, or 128, or maybe 256 or larger training
  examples. Batch norm works with mini-batch
  
  > **Comments:**
  > **username_4**: The punctuation needs some work.  
  > **username_3**: @username_4, I see your point.  
  > **username_0**: Hey, @username_3, my question specifically mentions these kinds of explanations. Experiments in the 2018 paper show that BatchNorm doesn't affect distributions in inner layers. Even more - **artificially** changing these distributions doesn't affect the performance much.  
  
  ---
  
  ### Answer by username_5 on 2022-01-06. Score: 0
  
  > When we are training deep neural Network gradient tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously. When we update, unexpected results can happen because many functions composed together are changed simultaneously using updates that were computed under the assumption that other function remains constant.This makes it very hard to choose an appropriate leaning rate, because the effects of an update to the parametrs of one layer strongly on all other layers.
  
  ***How does Batch Normalisation Help :***
  Batch Normalisation a layer which is added to any input or hidden layer in the neural network. Suppose H is the minitach of activations of the layer to normalize.
  
  The formula for normalizing H is :
  
  $\_H = \frac{H - Mean}{Standard Deviation}$
  
  Mean : Vector Containing Mean of each unit
  Standard Deviation : Vector Containing Mean of each unit
  
  At training time mean and sd are calculated and when we backpropogate through these operations for apply mean, sd and Normalize H. This means that gradient will never propose an operation that acts simply to increase the standard deviation and mean of hi, the normalization operation remove the effect of such an action and zero out its componenr in the gradient. Hence Batch Normalisation thus ensure no or slight covariance shift in the input to layer after Batch Normalisation and thus improving learning time as shown in the original paper mentioned in question.
  
  For more details : <https://www.deeplearningbook.org/contents/optimization.html>
  
  > **Comments:**
  > **username_6**: At least this "tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously." seems to be copied from [here](https://ai.stackexchange.com/q/20308/2444). Either edit this answer to clarify which parts you copied or this answer will be deleted.  
  > **username_5**: if you look at the reference we are referring to same reference. Nothing has been copied from the link :) The sentence you have highlighted is taken from reference.  
  > **username_6**: Yes, I understood that the reference is the same, but **you cannot just copy from a reference without explicitly showing us WHAT you copied**, i.e. you need to quote. To quote, you need to precede the text that you copied with `>`. Take a look at https://ai.stackexchange.com/help/referencing to know more about how to quote parts of other answers. Please, take the time to go through all your answers to quote things that you copied (from other sources) in order for them not to be deleted.  
  > **username_5**: Thanks i am new to this platform , will keep this things in mind  
  > **username_6**: Don't just put the reference at the end. Whenever you quote something, tell us immediately where you copied it from, which section and page, so that people can check that you're referencing a specific part of another source. That chapter 8 is a bit long.  
  
  ---
  
  ### Answer by username_7 on 2021-06-28. Score: -1
  
  It is a question with no simple answer.
  
  On one hand the BatchNormalization is unloved by some arguing it doesn't change the accuracy of neural networks or biased them.
  On the other hand, it is highly recommended by the other because it leads to better trained models with a larger scope of predictions and less chances of overflow.
  
  All I know for sure is that BN is really efficient on image classification. In fact, like the image categorization and classification soar this last years and that BN is a good practice in this field, it has spread to almost all DNNs.
  
  Not only is the BN not always used in the right purpose, but it is often used without taking into account several elements such as :
  
  * The layers between which apply BN
  * The initializer algorithms
  * The activation algorithms
  * etc
  
  For more computer sciences litterature "against" BN, I will let you look at the [H. Zhang et al paper](https://arxiv.org/abs/1901.09321) who has trained a DNN without BN and get good results.
  
  Some people use Gradient Clipping technique (R. Pascanu) instead of the BN in particular for RNNs
  
  I hope it will give you some answers !
  
  > **Comments:**
  > **username_8**: Thanks for the response. I'm following it as I added the bounty. I down voted it but don't take offence! The reason I did it is because I don't believe any part of it really addresses the question. Votes help others quickly identify the most relevant responses.  
  > **username_7**: Thanks for the feedback and the explanation. So I guess I didn't understand the question. Can you just reformulate it to let me understand it please ?  
  > **username_8**: I think your answer is more for a question like "Where is BN useful vs not?" or "Is BN always a good thing to use?". The question asks "What is the generally accepted reason for why BN works well?" Like if I ask "Why do conv kernels work well for images?" the answer is along the lines of: "Because it satisfies the inductive priors of translational invariance, feature hierarchies, and to some degree, scale invariance". Your answer doesn't talk about the properties of BN that make it successful. And in this case OP also wants the answer to be backed by authoritative sources  
  
  ---
  
  ### Answer by username_9 on 2021-04-16. Score: -2
  
  To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.
  
  > **Comments:**
  > **username_0**: Adding some _artificial_ noise to neural networks quite often increases their generalization performance.  
  > **username_9**: I was just thinking about the conditioning that BatchNorm performs and that also in some cases this process cutoff low intensity component. When a feature appears on a layer, it become evident since its intensity is very over average. When you "normalize",  all others components falls under threshold at least numerically. Then this "kind of noise" is removed. Another thing is adding *uniform noise*, which helps to search such a desirable feature and it is not removed by "normalization".  
  
  ---
  
  