Answers.Body: <p>It might be that the large labels dominate the loss value, so the model pays more attention to them. You could use an L1 loss rather than an L2 loss, such that the predictions get pulled towards each label equally rather than being pulled more strongly when the label is further away from the prediction.</p>
  
  <p>There may also be a data imbalance, i.e. you train on more large labels than small labels. This would also cause the model to pay more attention to becoming good at predicting the large labels. If this is the case, you can either collect more small labels or train more often on the small labels you have.</p>
  
  <p>Another possibility is that your labels are not evenly distributed. For example, the gaps between large data points may be larger than between small data points, so the small data points all look similar to the model. You can plot a histogram of your labels to find out and then transform your labels (e.g. using log) to make space them out more evenly.</p>
  
  <p>The general rules for training artificial neural networks apply. For example, normalize your inputs and outputs by subtracting the mean and divide them by their standard deviation, estimated across the training set. It also seem that your network is likely too deep and and not wide enough. I'd try 4x100 rather than 8x20 and add a small amount of weight decay when you see overfitting.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T03:35:07.517Z
Answers.LastActivityDate: 2020-04-28T03:35:07.517Z
Answers.UserDisplayName: danijar
Answers.Body: <p>Probably you could:</p>
  <ul>
  <li>Normalize your dataset</li>
  <li>Add Regularization</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T15:17:08.113Z
Answers.Text: Hi @Prateek! If possible, please add some supporting information to this answer to help other readers understand why the answer is correct. Thanks!
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-14T09:49:25.470Z
Answers.LastActivityDate: 2023-05-14T09:49:25.470Z
Answers.UserDisplayName: prat__
Answers.Body: <p>One option is <a href="https://en.wikipedia.org/wiki/Normalization_(statistics)" rel="nofollow noreferrer">normalizing</a> your data. In particular, min-max feature scaling to bring all values into the range [0,1] is particularly useful with gradient descent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:41:55.580Z
Answers.LastActivityDate: 2020-04-27T15:41:55.580Z
Answers.UserDisplayName: Brian Spiering
Answers.Body: <p>I think there is no special method for training the neural network in large datasets. But I can add some suggestion for you:</p>
  
  <p>1) Use the convolutional neural network for this dataset.</p>
  
  <p>2) You can use huber loss instead of squared loss and see what happens.</p>
  
  <p>3) See if you have enough small magnitude training data.</p>
  
  <p>Also, please define your problem in more detail (like what those images represent and what you want to predict, what are the features etc).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T06:22:23.013Z
Answers.LastActivityDate: 2020-04-27T06:22:23.013Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. <span class="math-container">$y = [0.014, 1.545, 10.232, 0.948, ...]$</span> corresponding to different points in time. The loss function in the fully connected neural network (input layer: 3 nodes, 8 inner layers: 20 nodes each, output layer: 1 node) is just the squared error between my prediction and the actual continuous data. It appears the neural network is able to learn the high magnitude data points relatively well (e.g. Figure 1 at time = 0.4422). But the smaller magnitude data points (e.g. Figure 2 at time = 1.1256) are quite poorly learned without any sharpness and I want to improve this. I've tried experimenting with different optimizers (e.g. mini-batch with Adam, full batch with L-BFGS), compared <code>reduce_mean</code> and <code>reduce_sum</code>, normalized the data in different ways (e.g. median, subtract the sample mean and divide by the standard deviation, divide the squared loss term by the actual data), and attempted to simply make the neural network deeper and train for a very long period of time (e.g. 7+ days). But after approximately 24 hours of training and the aforementioned tricks, I am not seeing any significant improvements in predicted outputs especially for the small magnitude data points.</p>
  
  <hr>
  
  <p><strong>Figure 1</strong></p>
  
  <p><a href="https://i.sstatic.net/NvLRi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NvLRi.png" alt="Figure 1 at time = 0.4422"></a></p>
  
  <hr>
  
  <p><strong>Figure 2</strong></p>
  
  <p><a href="https://i.sstatic.net/e64Lb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/e64Lb.png" alt="Figure 2 at time = 1.1256"></a></p>
  
  <hr>
  
  <p>Therefore, do you have any recommendations on how to improve training particularly when there are different data points of varying magnitude I am trying to learn? I believe <a href="https://stats.stackexchange.com/questions/221476/feature-varying-in-orders-of-magnitude">this</a> is a related question, but any explicit examples of implementations or techniques to handle varying orders of magnitude within a single large data set would be greatly appreciated.</p>
  
Tags: <neural-networks><tensorflow><training><optimization><batch-normalization>
Title: How to improve neural network training against a large data set of points with varying magnitude
UserDisplayName: Mathews24
ThreadText: # How to improve neural network training against a large data set of points with varying magnitude
  
  (Asked by: username_0 on 2020-04-22)
  
  I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$ corresponding to different points in time. The loss function in the fully connected neural network (input layer: 3 nodes, 8 inner layers: 20 nodes each, output layer: 1 node) is just the squared error between my prediction and the actual continuous data. It appears the neural network is able to learn the high magnitude data points relatively well (e.g. Figure 1 at time = 0.4422). But the smaller magnitude data points (e.g. Figure 2 at time = 1.1256) are quite poorly learned without any sharpness and I want to improve this. I've tried experimenting with different optimizers (e.g. mini-batch with Adam, full batch with L-BFGS), compared `reduce_mean` and `reduce_sum`, normalized the data in different ways (e.g. median, subtract the sample mean and divide by the standard deviation, divide the squared loss term by the actual data), and attempted to simply make the neural network deeper and train for a very long period of time (e.g. 7+ days). But after approximately 24 hours of training and the aforementioned tricks, I am not seeing any significant improvements in predicted outputs especially for the small magnitude data points.
  
  ---
  
  **Figure 1**
  
  [![Figure 1 at time = 0.4422](https://i.sstatic.net/NvLRi.png)](https://i.sstatic.net/NvLRi.png)
  
  ---
  
  **Figure 2**
  
  [![Figure 2 at time = 1.1256](https://i.sstatic.net/e64Lb.png)](https://i.sstatic.net/e64Lb.png)
  
  ---
  
  Therefore, do you have any recommendations on how to improve training particularly when there are different data points of varying magnitude I am trying to learn? I believe [this](https://stats.stackexchange.com/questions/221476/feature-varying-in-orders-of-magnitude) is a related question, but any explicit examples of implementations or techniques to handle varying orders of magnitude within a single large data set would be greatly appreciated.
  
  **Tags:** neural-networks, tensorflow, training, optimization, batch-normalization
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2020-04-28. Score: 0
  
  It might be that the large labels dominate the loss value, so the model pays more attention to them. You could use an L1 loss rather than an L2 loss, such that the predictions get pulled towards each label equally rather than being pulled more strongly when the label is further away from the prediction.
  
  There may also be a data imbalance, i.e. you train on more large labels than small labels. This would also cause the model to pay more attention to becoming good at predicting the large labels. If this is the case, you can either collect more small labels or train more often on the small labels you have.
  
  Another possibility is that your labels are not evenly distributed. For example, the gaps between large data points may be larger than between small data points, so the small data points all look similar to the model. You can plot a histogram of your labels to find out and then transform your labels (e.g. using log) to make space them out more evenly.
  
  The general rules for training artificial neural networks apply. For example, normalize your inputs and outputs by subtracting the mean and divide them by their standard deviation, estimated across the training set. It also seem that your network is likely too deep and and not wide enough. I'd try 4x100 rather than 8x20 and add a small amount of weight decay when you see overfitting.
  
  
  ---
  
  ### Answer by username_2 on 2023-05-14. Score: 0
  
  Probably you could:
  
  * Normalize your dataset
  * Add Regularization
  
  > **Comments:**
  > **username_3**: Hi @Prateek! If possible, please add some supporting information to this answer to help other readers understand why the answer is correct. Thanks!  
  
  ---
  
  ### Answer by username_4 on 2020-04-27. Score: 0
  
  One option is [normalizing](https://en.wikipedia.org/wiki/Normalization_(statistics)) your data. In particular, min-max feature scaling to bring all values into the range [0,1] is particularly useful with gradient descent.
  
  
  ---
  
  ### Answer by username_5 on 2020-04-27. Score: 0
  
  I think there is no special method for training the neural network in large datasets. But I can add some suggestion for you:
  
  1) Use the convolutional neural network for this dataset.
  
  2) You can use huber loss instead of squared loss and see what happens.
  
  3) See if you have enough small magnitude training data.
  
  Also, please define your problem in more detail (like what those images represent and what you want to predict, what are the features etc).
  
  
  ---
  
  