Answers.Body: <p>I have found a related part in</p>
  <blockquote>
  <p>C. C. Aggarwal. Neural Networks and Deep Learning: A Textbook. Springer International Publishing AG, 2018. isbn: 978-3-319-94462-3. doi: 10.1007/978-3-319-94463-0.</p>
  </blockquote>
  <p>about &quot;Additive preprocessing and mean-centering&quot; and &quot;Feature normalization&quot; on page 127. It says:</p>
  <blockquote>
  <p>It can be useful to mean-center the data in order to remove certain types of bias effects. Many algorithms in traditional machine learning (such as principal component analysis) also work with the assumption of mean-centered data. In such cases, a vector of column-wise means is subtracted from each data point. Mean-centering is often paired with standardization, which is discussed in the section of feature normalization.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-16T00:33:05.913Z
Answers.LastActivityDate: 2023-12-16T00:33:05.913Z
Answers.UserDisplayName: ha7ilm
Answers.Body: <p>The following paper is about time series and provides a good explanation.</p>
  <p>Passalis, N., Kanniainen, J., Gabbouj, M. et al. Forecasting Financial Time Series Using Robust Deep Adaptive Input Normalization. J Sign Process Syst 93, 1235–1251 (2021). <a href="https://doi.org/10.1007/s11265-020-01624-0" rel="nofollow noreferrer">https://doi.org/10.1007/s11265-020-01624-0</a>
  <a href="https://deepfinance.csd.auth.gr/wp-content/uploads/2021/05/Forecasting-Financial-Time-Series-using-Robust-Deep-Adaptive-Input-Normalization.pdf" rel="nofollow noreferrer">PDF</a></p>
  <blockquote>
  <p>One question that naturally arises from the aforementioned
  observations is why normalization seems to affect the performance of
  DL algorithms to this extent. To better understand this we need to
  consider two different phenomena that often arise. First, the employed
  normalization scheme significantly affects the generalization of the
  model on unknown data. For example, when the raw price is used as a
  feature, then a DL model operates on an fully unknown region of the
  input space when the price exceeds the range of price values used
  during the training. Methods such as sample-based standardization and
  instance normalization [14] can mitigate the effects of this to some
  extent, leading to improved generalization results. It is also worth
  noting that recent literature demonstrated that applying the
  appropriate input pre-processing Schemes, such as including the
  normalized first order differences, emphasizing on specific
  characteristics of the data, e.g., zero-crossing rate, can lead to
  significant improvements in forecasting precision and trading
  performance [18, 19], further highlighting the importance of
  appropriately pre-processing the input time series. The second reason
  is intrinsic to the nature of the gradient descent-based optimization
  that is employed for training DL models. Even though the input
  features carry the same amount of information regardless of the
  scaling applied to the individual features, the gradient descent’s
  updates that are used for training the model are affected by the scale
  of the input features. Therefore, features with large magnitude have a
  larger effect on the direction of the descent, often leading the model
  toward different local minima than a model for which all the input
  feature were normalized in the same scale. The normalization therefore
  acts as a way to ensure that each feature will have “equal” chances
  for affecting the optimization.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-17T03:02:15.100Z
Answers.LastActivityDate: 2023-12-17T03:02:15.100Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>We can read that the normalization of the data that is input to a neural network is important and is considered as a best practice, for example <a href="https://stackoverflow.com/questions/4674623/why-do-we-have-to-normalize-the-input-for-an-artificial-neural-network">SO #1</a>, <a href="https://stats.stackexchange.com/questions/458579/should-i-normalize-all-data-prior-feeding-the-neural-network-models">SO #2</a> and many other places. It looks to me that there is no consensus on why it works, but there are a wide variety of possible reasons, including that the initialization of the weights is assuming normalized data, and that the gradient descent might become unstable if the input variables are scaled very differently (see <a href="https://stats.stackexchange.com/a/437848">here</a>).
  I myself have also experienced in an application that applying Z-score normalization was essential in order to achieve good results. On the other hand, my understanding is that it is not theoretically proven that normalization will always allow to produce better results than without it, and it is just one of the techniques in the engineer's toolbox to try.</p>
  <p><strong>Now the question is, which paper to cite about that normalization of inputs is a best practice?</strong></p>
  <p>I tried to look for a good reference for it but I have not found a really good one. For example, I cannot find this in particular in the <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Goodfellow book</a>. There is a lot about batch normalization (i.e., that is often applied in between layers, and the normalization parameters are learned), and some computer vision related normalizations, but not this. The <a href="https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">Glorot paper</a> does not mention normalization in this sense either (what they normalize is the initial weights). On the other hand, this technique is there in quite some blog posts and tutorials, e.g., <a href="https://medium.com/analytics-vidhya/deep-learning-basics-input-normalization-670735d3a832" rel="nofollow noreferrer">this one</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-17T17:42:55.040Z
Comments.Text: If this is on-topic, I don't think we have yet agreed on this. It looks like you're doing research and ask us to help you do it. We should probably discuss this on meta. Please, open a post there to ask about if this question is on-topic: https://ai.meta.stackexchange.com/questions/ask
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-17T20:24:31.053Z
Comments.Text: I’m voting to close this question because I think it's off-topic.
Comments.UserDisplayName: nbro
Tags: <neural-networks><reference-request><normalisation>
Title: Paper to cite about normalizing the inputs to a neural network
UserDisplayName: ha7ilm
ThreadText: # Paper to cite about normalizing the inputs to a neural network
  
  (Asked by: username_0 on 2023-12-15)
  
  We can read that the normalization of the data that is input to a neural network is important and is considered as a best practice, for example [SO #1](https://stackoverflow.com/questions/4674623/why-do-we-have-to-normalize-the-input-for-an-artificial-neural-network), [SO #2](https://stats.stackexchange.com/questions/458579/should-i-normalize-all-data-prior-feeding-the-neural-network-models) and many other places. It looks to me that there is no consensus on why it works, but there are a wide variety of possible reasons, including that the initialization of the weights is assuming normalized data, and that the gradient descent might become unstable if the input variables are scaled very differently (see [here](https://stats.stackexchange.com/a/437848)).
  I myself have also experienced in an application that applying Z-score normalization was essential in order to achieve good results. On the other hand, my understanding is that it is not theoretically proven that normalization will always allow to produce better results than without it, and it is just one of the techniques in the engineer's toolbox to try.
  
  **Now the question is, which paper to cite about that normalization of inputs is a best practice?**
  
  I tried to look for a good reference for it but I have not found a really good one. For example, I cannot find this in particular in the [Goodfellow book](https://www.deeplearningbook.org/). There is a lot about batch normalization (i.e., that is often applied in between layers, and the normalization parameters are learned), and some computer vision related normalizations, but not this. The [Glorot paper](https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) does not mention normalization in this sense either (what they normalize is the initial weights). On the other hand, this technique is there in quite some blog posts and tutorials, e.g., [this one](https://medium.com/analytics-vidhya/deep-learning-basics-input-normalization-670735d3a832).
  
  **Tags:** neural-networks, reference-request, normalisation
  
  > **Comments:**
  > **username_1**: If this is on-topic, I don't think we have yet agreed on this. It looks like you're doing research and ask us to help you do it. We should probably discuss this on meta. Please, open a post there to ask about if this question is on-topic: https://ai.meta.stackexchange.com/questions/ask  
  > **username_1**: I’m voting to close this question because I think it's off-topic.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-12-16. Score: 1
  
  I have found a related part in
  
  > C. C. Aggarwal. Neural Networks and Deep Learning: A Textbook. Springer International Publishing AG, 2018. isbn: 978-3-319-94462-3. doi: 10.1007/978-3-319-94463-0.
  
  about "Additive preprocessing and mean-centering" and "Feature normalization" on page 127. It says:
  
  > It can be useful to mean-center the data in order to remove certain types of bias effects. Many algorithms in traditional machine learning (such as principal component analysis) also work with the assumption of mean-centered data. In such cases, a vector of column-wise means is subtracted from each data point. Mean-centering is often paired with standardization, which is discussed in the section of feature normalization.
  
  
  ---
  
  ### Answer by username_2 on 2023-12-17. Score: 0
  
  The following paper is about time series and provides a good explanation.
  
  Passalis, N., Kanniainen, J., Gabbouj, M. et al. Forecasting Financial Time Series Using Robust Deep Adaptive Input Normalization. J Sign Process Syst 93, 1235–1251 (2021). <https://doi.org/10.1007/s11265-020-01624-0>
  [PDF](https://deepfinance.csd.auth.gr/wp-content/uploads/2021/05/Forecasting-Financial-Time-Series-using-Robust-Deep-Adaptive-Input-Normalization.pdf)
  
  > One question that naturally arises from the aforementioned
  > observations is why normalization seems to affect the performance of
  > DL algorithms to this extent. To better understand this we need to
  > consider two different phenomena that often arise. First, the employed
  > normalization scheme significantly affects the generalization of the
  > model on unknown data. For example, when the raw price is used as a
  > feature, then a DL model operates on an fully unknown region of the
  > input space when the price exceeds the range of price values used
  > during the training. Methods such as sample-based standardization and
  > instance normalization [14] can mitigate the effects of this to some
  > extent, leading to improved generalization results. It is also worth
  > noting that recent literature demonstrated that applying the
  > appropriate input pre-processing Schemes, such as including the
  > normalized first order differences, emphasizing on specific
  > characteristics of the data, e.g., zero-crossing rate, can lead to
  > significant improvements in forecasting precision and trading
  > performance [18, 19], further highlighting the importance of
  > appropriately pre-processing the input time series. The second reason
  > is intrinsic to the nature of the gradient descent-based optimization
  > that is employed for training DL models. Even though the input
  > features carry the same amount of information regardless of the
  > scaling applied to the individual features, the gradient descent’s
  > updates that are used for training the model are affected by the scale
  > of the input features. Therefore, features with large magnitude have a
  > larger effect on the direction of the descent, often leading the model
  > toward different local minima than a model for which all the input
  > feature were normalized in the same scale. The normalization therefore
  > acts as a way to ensure that each feature will have “equal” chances
  > for affecting the optimization.
  
  
  ---
  
  