Answers.Body: <p>The <a href="https://en.wikipedia.org/wiki/Central_limit_theorem" rel="nofollow noreferrer">central limit theorem</a> tells us that the error in an estimate of the mean or standard deviation of a dataset will decline as <span class="math-container">$\frac{1}{\sqrt{n}}$</span>, where <span class="math-container">$n$</span> is the number of samples taken at random from the set, and combined together to compute the mean and deviation.</p>
  
  <p>If you select at random, for example, <span class="math-container">$10^6$</span> examples (probably a few megabytes), then the mean and standard deviation you compute from those examples will be within <span class="math-container">$\frac{1}{sqrt(10^6)} = \frac{1}{10^3}$</span> of the "true" answer. That's one part in 1,000, which is certainly accurate enough to use for re-scaling the dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-04T16:44:50.247Z
Answers.Text: I was thinking along the same lines, To use certain number of observation for getting a ball park number, and now I could also quantity the error! But isn't there any more elegant solution to this?
Answers.UserDisplayName: thisisbhavin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-04T16:53:30.663Z
Answers.Text: You can also calculate mean and standard deviation using single-pass "streaming" approaches. Books like http://numerical.recipes/ cover these algorithms, and many more statistical algorithms for single-passes through batches of a dataset. However, the central limit theorem tells us these will probably not improve your accuracy enough to be worthwhile. Why spend many minutes (or perhaps hours) reading through all the data for a 0.1% improvement in a statistic you use only for normalization (which doesn't _need_ to be that accurate)?
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-04T23:11:57.763Z
Answers.Text: Indeed, its better to invest that time doing something  else that could actually help improve accuracy. Thank you
Answers.UserDisplayName: thisisbhavin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-04T16:15:28.047Z
Answers.LastActivityDate: 2019-11-04T16:15:28.047Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>One way could be to first iterate over the dataset in batches, just to get the mean and sd. Then when running training, use the true population parameters obtained before.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-03T15:12:29.220Z
Answers.LastActivityDate: 2019-11-03T15:12:29.220Z
Answers.UserDisplayName: pkapka
Body: <p>I have a structured dataset of around 100 gigs, and I am using DNN for classification in TF 2.0. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.</p>
  
  <p>Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have been reading TensorFlow docs to get info on how to normalize features when reading data in batches. But, couldn't find one. though I found this <a href="https://www.tensorflow.org/tutorials/load_data/csv" rel="nofollow noreferrer">article</a>, it is only for the case where entire data can be loaded in memory.</p>
  
  <p>So, If any of you have worked on creating such a TensorFlow data pipeline for normalizing input features while loading data in batches and training model, It would be helpful.</p>
  
Tags: <neural-networks><deep-learning><tensorflow><datasets>
Title: TensorFlow 2.0 - Normalizing input to DNN (on structured data)
UserDisplayName: thisisbhavin
ThreadText: # TensorFlow 2.0 - Normalizing input to DNN (on structured data)
  
  (Asked by: username_0 on 2019-11-03)
  
  I have a structured dataset of around 100 gigs, and I am using DNN for classification in TF 2.0. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.
  
  Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have been reading TensorFlow docs to get info on how to normalize features when reading data in batches. But, couldn't find one. though I found this [article](https://www.tensorflow.org/tutorials/load_data/csv), it is only for the case where entire data can be loaded in memory.
  
  So, If any of you have worked on creating such a TensorFlow data pipeline for normalizing input features while loading data in batches and training model, It would be helpful.
  
  **Tags:** neural-networks, deep-learning, tensorflow, datasets
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-11-04. Score: 0
  
  The [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) tells us that the error in an estimate of the mean or standard deviation of a dataset will decline as $\frac{1}{\sqrt{n}}$, where $n$ is the number of samples taken at random from the set, and combined together to compute the mean and deviation.
  
  If you select at random, for example, $10^6$ examples (probably a few megabytes), then the mean and standard deviation you compute from those examples will be within $\frac{1}{sqrt(10^6)} = \frac{1}{10^3}$ of the "true" answer. That's one part in 1,000, which is certainly accurate enough to use for re-scaling the dataset.
  
  > **Comments:**
  > **username_0**: I was thinking along the same lines, To use certain number of observation for getting a ball park number, and now I could also quantity the error! But isn't there any more elegant solution to this?  
  > **username_1**: You can also calculate mean and standard deviation using single-pass "streaming" approaches. Books like http://numerical.recipes/ cover these algorithms, and many more statistical algorithms for single-passes through batches of a dataset. However, the central limit theorem tells us these will probably not improve your accuracy enough to be worthwhile. Why spend many minutes (or perhaps hours) reading through all the data for a 0.1% improvement in a statistic you use only for normalization (which doesn't _need_ to be that accurate)?  
  > **username_0**: Indeed, its better to invest that time doing something  else that could actually help improve accuracy. Thank you  
  
  ---
  
  ### Answer by username_2 on 2019-11-03. Score: 0
  
  One way could be to first iterate over the dataset in batches, just to get the mean and sd. Then when running training, use the true population parameters obtained before.
  
  
  ---
  
  