Answers.Body: <p>Neural networks learn badly with large input ranges. Scale your inputs to a smaller range e.g. -2 to 2, and convert to/from this range to represent your function interval consistently.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-14T22:57:46.240Z
Answers.Text: Are you sure just linearly putting values in another range conduces better results? It seems that there is nothing qualitatively different from the two.
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-14T23:39:41.560Z
Answers.Text: @Daniel: Yes I am sure. It can make a huge difference in practice. *Always* check the range of your inputs to a NN and scale them if necessary. The numerically ideal scale is around mean 0, standard deviation 1. There is some flexibility around that, but if you have inputs with values an order of magnitude or more different to that, the NN learning will suffer.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-15T21:56:44.370Z
Answers.Text: @NeilSlater Oh, I thought you meant you meant just scaling the values between an interval and not normalizing them.
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-09T11:54:37.387Z
Answers.LastActivityDate: 2018-04-09T11:54:37.387Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I only have one good news... There is nothing wrong with your code. Neural networks tend to do that. Especially with a really complex function.</p>
  <ul>
  <li>Increasing the amount of neurons will not decrease how the error is distributed.</li>
  <li>There are better loss functions for each case but is not a really effective solution.</li>
  <li>Neural networks are really good managing noise. So, they are good ignoring minorities. It's a common expression &quot;ANN are racist&quot;.</li>
  </ul>
  <p>I recommend you to deploy a histogram <em>DataSet</em> vs <em>Output Value</em>. To see if you have too much more data from the central region than in the frontier.
  If you can generate more data at will. Generate more values in the specific zones with more errors.<br />
  This will increase the error and force the backpropagation algorithm to improve in that area.<br />
  More information on your optimization algorithm may be useful. But, like I said, everything seems perfectly normal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-09T13:13:56.080Z
Answers.LastActivityDate: 2020-10-10T16:11:22.370Z
Answers.LastEditDate: 2020-10-10T16:11:22.370Z
Answers.UserDisplayName: Joaquin Torrens
Body: <p>The problem to solve is non-linear regression of a non-linear function. My actual problem is to model the function "find the max over many quadratic forms": <code>max(w.H.T * Q * w)</code>, but to get started and to learn more about neural networks, I created a toy example for a non-linear regression task, using Pytorch. 
  The problem is that the network never learns the function in a satisfactory way, even though my model is quite large with multiple layers (see below). Or is it not large enough or too large? How can the network be improved or maybe even simplified to get a much smaller training error?</p>
  
  <p>I experimented with different network architectures, but the result is never satisfactory. Usually, the error is quite small within the input interval around 0, but the network is not able to get good weights for the regions at the boundary of the interval (see plots below). The loss does not improve after a certain number of epochs.  I could generate even more training data, but I have not yet understood completely, how the training can be improved (tuning parameters such as batch size, amount of data, number of layers, normalizing input (output?) data,  number of neurons, epochs, etc.)</p>
  
  <p>My neural network has 8 layers with the following number of neurons: <code>1, 80, 70, 60, 40, 40, 20, 1</code>.</p>
  
  <p>For the moment, I do not care too much about overfitting, my goal is to understand, why a certain network architecture/certain hyperparameters need to be chosen. Of course, avoiding overfitting at the same time would be a bonus.</p>
  
  <p>I am especially interested in using neural networks for regression tasks or as function approximators. In principle, my problem should be able to be approximated to arbitrary accuracy by a single layer neural network, according to the universal approximation theorem, isn’t this correct?</p>
  
  <p><a href="https://i.sstatic.net/oqXXN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oqXXN.png" alt="Difference between the trained model and original data"></a></p>
  
  <p><a href="https://i.sstatic.net/l2U6j.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/l2U6j.png" alt="Loss value vs iterations"></a></p>
  
  <p><a href="https://i.sstatic.net/sHAct.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sHAct.png" alt="Difference between the trained model and original data "></a></p>
  
Tags: <neural-networks><machine-learning><deep-learning><pytorch>
Title: Why isn't my model learning satisfactorily?
UserDisplayName: BPresent
ThreadText: # Why isn't my model learning satisfactorily?
  
  (Asked by: username_0 on 2018-04-09)
  
  The problem to solve is non-linear regression of a non-linear function. My actual problem is to model the function "find the max over many quadratic forms": `max(w.H.T * Q * w)`, but to get started and to learn more about neural networks, I created a toy example for a non-linear regression task, using Pytorch.
  The problem is that the network never learns the function in a satisfactory way, even though my model is quite large with multiple layers (see below). Or is it not large enough or too large? How can the network be improved or maybe even simplified to get a much smaller training error?
  
  I experimented with different network architectures, but the result is never satisfactory. Usually, the error is quite small within the input interval around 0, but the network is not able to get good weights for the regions at the boundary of the interval (see plots below). The loss does not improve after a certain number of epochs. I could generate even more training data, but I have not yet understood completely, how the training can be improved (tuning parameters such as batch size, amount of data, number of layers, normalizing input (output?) data, number of neurons, epochs, etc.)
  
  My neural network has 8 layers with the following number of neurons: `1, 80, 70, 60, 40, 40, 20, 1`.
  
  For the moment, I do not care too much about overfitting, my goal is to understand, why a certain network architecture/certain hyperparameters need to be chosen. Of course, avoiding overfitting at the same time would be a bonus.
  
  I am especially interested in using neural networks for regression tasks or as function approximators. In principle, my problem should be able to be approximated to arbitrary accuracy by a single layer neural network, according to the universal approximation theorem, isn’t this correct?
  
  [![Difference between the trained model and original data](https://i.sstatic.net/oqXXN.png)](https://i.sstatic.net/oqXXN.png)
  
  [![Loss value vs iterations](https://i.sstatic.net/l2U6j.png)](https://i.sstatic.net/l2U6j.png)
  
  [![Difference between the trained model and original data ](https://i.sstatic.net/sHAct.png)](https://i.sstatic.net/sHAct.png)
  
  **Tags:** neural-networks, machine-learning, deep-learning, pytorch
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-04-09. Score: 1 | Accepted answer
  
  Neural networks learn badly with large input ranges. Scale your inputs to a smaller range e.g. -2 to 2, and convert to/from this range to represent your function interval consistently.
  
  > **Comments:**
  > **username_2**: Are you sure just linearly putting values in another range conduces better results? It seems that there is nothing qualitatively different from the two.  
  > **username_1**: @username_2: Yes I am sure. It can make a huge difference in practice. *Always* check the range of your inputs to a NN and scale them if necessary. The numerically ideal scale is around mean 0, standard deviation 1. There is some flexibility around that, but if you have inputs with values an order of magnitude or more different to that, the NN learning will suffer.  
  > **username_2**: @username_1 Oh, I thought you meant you meant just scaling the values between an interval and not normalizing them.  
  
  ---
  
  ### Answer by username_3 on 2020-10-09. Score: 0
  
  I only have one good news... There is nothing wrong with your code. Neural networks tend to do that. Especially with a really complex function.
  
  * Increasing the amount of neurons will not decrease how the error is distributed.
  * There are better loss functions for each case but is not a really effective solution.
  * Neural networks are really good managing noise. So, they are good ignoring minorities. It's a common expression "ANN are racist".
  
  I recommend you to deploy a histogram *DataSet* vs *Output Value*. To see if you have too much more data from the central region than in the frontier.
  If you can generate more data at will. Generate more values in the specific zones with more errors.  
  This will increase the error and force the backpropagation algorithm to improve in that area.  
  More information on your optimization algorithm may be useful. But, like I said, everything seems perfectly normal.
  
  
  ---
  
  