Answers.Body: <blockquote>
  <p>When using neural networks (NNs), we often normalized the inputs. I
  think this is done to equally capture the changes in any input
  feature, that is, if any feature takes huge values and other features
  take small values, we don't want the NN not to be able to &quot;see&quot; the change
  in the smaller value.</p>
  <p>However, what if we cause the NN to become insensitive to the input,
  that is, the NN is not able to identify changes in the input
  because the changes are too small?</p>
  </blockquote>
  <p>We <strong>don't</strong> normalize the input to make the model less sensitive to small changes in the input (theoretically, given the correct optimization strategy, the model will learn to approximate the smaller-ranged input as well).</p>
  <p>An example of this would be Convolutional Neural Networks. Traditionally, images were represented with integer values ranging from <span class="math-container">$0$</span> to <span class="math-container">$255$</span>. This means that a given pixel could have <strong>only <span class="math-container">$256$</span> distinct values</strong>. However, assuming we normalize the input, let's say to <span class="math-container">$[0, 1]$</span>, this gives the pixel a whole range of values to occupy, making the input <em>more sensitive to changes.</em></p>
  <p>Instead, normalization is done to help with the model's <strong>convergence</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T00:11:44.470Z
Answers.Text: I do not disagree with you, but what you are describing is *scaling*, not *normalizing*.  Normalizing the input implies that you are forcing it to take more of a Gaussian distribution's shape.  Scaling is simply forcing the values to be in a specific range.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T05:50:10.220Z
Answers.Text: Hi, in the context of data science *normalization* means bringing the inputs to take a range of $[0, 1]$ (e.g. see [here](https://stats.stackexchange.com/a/10298/119015)). *Normalization* is just a type of feature scaling.
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T13:00:35.887Z
Answers.Text: I don't have time to find a better reference for you at the moment, but I'd refer you to https://towardsai.net/p/data-science/scaling-vs-normalizing-data-5c3514887a84 for now
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T14:03:28.477Z
Answers.Text: Under different contexts, the term "normalization" can mean different things Most people in data science when saying normalization mean scaling to $[0, 1]$. You could do a google search on "normalization data science", "feature scaling" or something like that to see what I'm talking about.
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T14:07:15.530Z
Answers.Text: I totally get what you're saying.  It's just not my experience.  The distinction is pretty important because when you normalize (gaussian) your training data, it becomes very difficult to apply the same transformation to reality.  Scaling, however, is trivial. :)
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T14:11:24.410Z
Answers.Text: As examples, I'd point you to The Deep Learning Book (Goodfellow,et al) pages 73, 138, 183, 184, 262, and others.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-06T22:01:23.503Z
Answers.Text: Scaling the input from 0-255 to 0-1 doesn't increase the sensitivity, if you were to stick with integer numbers. Arguably you should extend the scale to 16-bit integers, or just use floats (what you are implying).
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-24T11:52:44.973Z
Answers.LastActivityDate: 2020-11-15T18:39:57.427Z
Answers.LastEditDate: 2020-11-15T18:39:57.427Z
Answers.UserDisplayName: Djib2011
Body: <p>When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to &quot;see&quot; the change in the smaller value.</p>
  <p>However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T00:10:19.713Z
Comments.Text: You should also make sure you aren't confusing *scaling* the inputs with *normalizing* the inputs.  These are very different things.
Comments.UserDisplayName: David Hoelzer
Tags: <neural-networks><deep-learning><data-preprocessing><normalisation>
Title: Could the normalisation of the inputs make the neural network insensitive to changes in the inputs?
UserDisplayName: pranav
ThreadText: # Could the normalisation of the inputs make the neural network insensitive to changes in the inputs?
  
  (Asked by: username_0 on 2019-06-24)
  
  When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.
  
  However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?
  
  **Tags:** neural-networks, deep-learning, data-preprocessing, normalisation
  > **Comments:**
  > **username_1**: You should also make sure you aren't confusing *scaling* the inputs with *normalizing* the inputs.  These are very different things.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-06-24. Score: 0
  
  > When using neural networks (NNs), we often normalized the inputs. I
  > think this is done to equally capture the changes in any input
  > feature, that is, if any feature takes huge values and other features
  > take small values, we don't want the NN not to be able to "see" the change
  > in the smaller value.
  >
  > However, what if we cause the NN to become insensitive to the input,
  > that is, the NN is not able to identify changes in the input
  > because the changes are too small?
  
  We **don't** normalize the input to make the model less sensitive to small changes in the input (theoretically, given the correct optimization strategy, the model will learn to approximate the smaller-ranged input as well).
  
  An example of this would be Convolutional Neural Networks. Traditionally, images were represented with integer values ranging from $0$ to $255$. This means that a given pixel could have **only $256$ distinct values**. However, assuming we normalize the input, let's say to $[0, 1]$, this gives the pixel a whole range of values to occupy, making the input *more sensitive to changes.*
  
  Instead, normalization is done to help with the model's **convergence**.
  
  > **Comments:**
  > **username_1**: I do not disagree with you, but what you are describing is *scaling*, not *normalizing*.  Normalizing the input implies that you are forcing it to take more of a Gaussian distribution's shape.  Scaling is simply forcing the values to be in a specific range.  
  > **username_2**: Hi, in the context of data science *normalization* means bringing the inputs to take a range of $[0, 1]$ (e.g. see [here](https://stats.stackexchange.com/a/10298/119015)). *Normalization* is just a type of feature scaling.  
  > **username_1**: I don't have time to find a better reference for you at the moment, but I'd refer you to https://towardsai.net/p/data-science/scaling-vs-normalizing-data-5c3514887a84 for now  
  > **username_2**: Under different contexts, the term "normalization" can mean different things Most people in data science when saying normalization mean scaling to $[0, 1]$. You could do a google search on "normalization data science", "feature scaling" or something like that to see what I'm talking about.  
  > **username_1**: I totally get what you're saying.  It's just not my experience.  The distinction is pretty important because when you normalize (gaussian) your training data, it becomes very difficult to apply the same transformation to reality.  Scaling, however, is trivial. :)  
  > **username_1**: As examples, I'd point you to The Deep Learning Book (Goodfellow,et al) pages 73, 138, 183, 184, 262, and others.  
  > **username_3**: Scaling the input from 0-255 to 0-1 doesn't increase the sensitivity, if you were to stick with integer numbers. Arguably you should extend the scale to 16-bit integers, or just use floats (what you are implying).  
  
  ---
  
  