Answers.Body: <p>I answered a similar question earlier and here is a piece of my answer that i think covers your question:</p>
  
  <p>Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>) but lately has been tied to the optimization process (<a href="https://arxiv.org/pdf/1805.11604.pdf" rel="nofollow noreferrer">How Does Batch Normalization Help Optimization?</a>). </p>
  
  <p>This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized or how it will effect your network, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given your initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: <a href="https://arxiv.org/abs/1901.09321" rel="nofollow noreferrer">Fixup Initialization</a>). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-17T22:18:27.970Z
Answers.LastActivityDate: 2019-07-17T22:18:27.970Z
Answers.UserDisplayName: mshlis
Body: <p>I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale"</p>
  
  <p>How can <code>x-m/s</code> make the scale of images the same? Please explain to me the maths. Also, take MNIST dataset for example &amp; illustration.</p>
  
Tags: <neural-networks><convolutional-neural-networks><data-preprocessing>
Title: Why do we normalize data in a deep neural network?
UserDisplayName: abhimanyuaryan
ThreadText: # Why do we normalize data in a deep neural network?
  
  (Asked by: username_0 on 2019-07-17)
  
  I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale"
  
  How can `x-m/s` make the scale of images the same? Please explain to me the maths. Also, take MNIST dataset for example & illustration.
  
  **Tags:** neural-networks, convolutional-neural-networks, data-preprocessing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-17. Score: 3
  
  I answered a similar question earlier and here is a piece of my answer that i think covers your question:
  
  Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167)) but lately has been tied to the optimization process ([How Does Batch Normalization Help Optimization?](https://arxiv.org/pdf/1805.11604.pdf)).
  
  This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized or how it will effect your network, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given your initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: [Fixup Initialization](https://arxiv.org/abs/1901.09321)).
  
  
  ---
  
  