Answers.Body: <p>It depends on how your data and prediction task are structured.
  If you normalize per sample, you lose all relative information between samples. On the other hand, if your data contains a lot of samples of different magnitude,normalizing per sample might help achieving more stable gradients and training. It might also help with out-of-distribution test data.</p>
  <p>Now if your prediction task is only dependent on the information present within one sample, and does not really depend on how a single sample stands in comparison to other samples, then normalizing per sample is fine. If this is not the case and the relative information is important, you could either add the normalization parameters as an additional feature, or resort to classical normalization.</p>
  <p>As mentioned in the other <a href="https://ai.stackexchange.com/a/38324/64988">answer</a>, the only way to know is a proper evaluation routine like cross validation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-14T10:46:03.567Z
Answers.LastActivityDate: 2022-12-14T10:46:03.567Z
Answers.UserDisplayName: besterma
Answers.Body: <blockquote>
  <p>Would it change the neural network for the better or worse?</p>
  </blockquote>
  <p>Most likely worse, because it will make the normalisation parameters noisy (varying per sample). In the case of using strictly tabular data where there is no repeated element type within a data sample, you don't really have anything to normalise by, although you could ignore the difference in data types and normalise across each row of data. I would expect this to have terrible performance in most cases.</p>
  <p>However, in your case, you have a lot of repeated elements, so this noise <em>may</em> be within reasonable limits. Also, it <em>might</em> turn out that predicting output from input normalised in this way suits your problem domain.</p>
  <p>The only way to find out is to try multiple normalisation approaches and use cross validation scores to see which one performs best.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-14T10:10:20.963Z
Answers.LastActivityDate: 2022-12-14T10:10:20.963Z
Answers.UserDisplayName: Neil Slater
Body: <p>The standard method is to normalize the entire dataset (the training part) then send it to the model to train on. However I’ve noticed that in this manner the model doesn’t really work well when dealing with values outside the range it was trained on.</p>
  <p>So how about normalizing each sample between a fixed range, say 0 to 1 and then sending them in.</p>
  <p>Of course the testing data and the values to predict on would also be normalized in the same way.</p>
  <p>Would it change the neural network for the better or worse?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-14T09:41:32.397Z
Comments.Text: What kind of samples are you dealing with? E.g. images, or tabular data with one value in each sample for each column, or something else?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-14T09:45:31.340Z
Comments.Text: Its a 7-featured dataset, with 4 hours minute-to-minute dataset of everyday for the last 5 years. I am inputing 2-3 hours of data sequence to get the remaining 2-1 hour sequence. What would happen if I rather normalize each sample individually here rather than the entire dataset?
Comments.UserDisplayName: Maks
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-14T09:48:36.920Z
Comments.Text: So a single training sample for you consists of a sequence of 120+ 7-dimensional vectors as the input and 60+ 7-dimensional vectors as output?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-14T09:59:12.857Z
Comments.Text: Yes. That is what I am doing currently. Experimenting with different configurations of input. So what do you think would happen if each sample is normalized on its own?
Comments.UserDisplayName: Maks
Tags: <neural-networks><normalisation>
Title: What if each sample was normalized on its own before sending them to the neural network?
UserDisplayName: Maks
ThreadText: # What if each sample was normalized on its own before sending them to the neural network?
  
  (Asked by: username_0 on 2022-12-14)
  
  The standard method is to normalize the entire dataset (the training part) then send it to the model to train on. However I’ve noticed that in this manner the model doesn’t really work well when dealing with values outside the range it was trained on.
  
  So how about normalizing each sample between a fixed range, say 0 to 1 and then sending them in.
  
  Of course the testing data and the values to predict on would also be normalized in the same way.
  
  Would it change the neural network for the better or worse?
  
  **Tags:** neural-networks, normalisation
  
  > **Comments:**
  > **username_1**: What kind of samples are you dealing with? E.g. images, or tabular data with one value in each sample for each column, or something else?  
  > **username_0**: Its a 7-featured dataset, with 4 hours minute-to-minute dataset of everyday for the last 5 years. I am inputing 2-3 hours of data sequence to get the remaining 2-1 hour sequence. What would happen if I rather normalize each sample individually here rather than the entire dataset?  
  > **username_1**: So a single training sample for you consists of a sequence of 120+ 7-dimensional vectors as the input and 60+ 7-dimensional vectors as output?  
  > **username_0**: Yes. That is what I am doing currently. Experimenting with different configurations of input. So what do you think would happen if each sample is normalized on its own?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-12-14. Score: 3
  
  It depends on how your data and prediction task are structured.
  If you normalize per sample, you lose all relative information between samples. On the other hand, if your data contains a lot of samples of different magnitude,normalizing per sample might help achieving more stable gradients and training. It might also help with out-of-distribution test data.
  
  Now if your prediction task is only dependent on the information present within one sample, and does not really depend on how a single sample stands in comparison to other samples, then normalizing per sample is fine. If this is not the case and the relative information is important, you could either add the normalization parameters as an additional feature, or resort to classical normalization.
  
  As mentioned in the other [answer](https://ai.stackexchange.com/a/38324/64988), the only way to know is a proper evaluation routine like cross validation.
  
  
  ---
  
  ### Answer by username_1 on 2022-12-14. Score: 0
  
  > Would it change the neural network for the better or worse?
  
  Most likely worse, because it will make the normalisation parameters noisy (varying per sample). In the case of using strictly tabular data where there is no repeated element type within a data sample, you don't really have anything to normalise by, although you could ignore the difference in data types and normalise across each row of data. I would expect this to have terrible performance in most cases.
  
  However, in your case, you have a lot of repeated elements, so this noise *may* be within reasonable limits. Also, it *might* turn out that predicting output from input normalised in this way suits your problem domain.
  
  The only way to find out is to try multiple normalisation approaches and use cross validation scores to see which one performs best.
  
  
  ---
  
  