Answers.Body: <p>You could try to replace MSE with the recently proposed <a href="https://arxiv.org/pdf/1701.03077.pdf" rel="nofollow noreferrer">General and Adaptive Robust Loss Function</a>, a generalization of several losses among which l2 norm (which aside from a square root is equal to MSE). The neat property of this loss is that it uses a dynamic parameter <span class="math-container">$\alpha$</span> which makes the loss behaving differently, that can be learned during training. In this way the loss itself adapt to specific outliers in your data (the authors made a <a href="https://www.youtube.com/watch?v=BmNKbnF69eY" rel="nofollow noreferrer">video</a> with great visuals to explain this point).</p>
  <p>I suspect at first glance (i.e. without data examples) that the problem your dealing with is that steep points are treated by your network as outliers, i.e. your model learn to generalize a smooth function cause the function is smooth most of the time, and surely also cause the amount of parameters of your network is really small (so it has a hard time fitting local irregular patterns). If it true that your function is smooth most of the time than oversampling steep points would be a mistake, cause that would mean training your network on a non representative sample of points, hence if this adaptive loss doesn't improve much your results I would just make the network larger.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-26T15:12:15.243Z
Answers.Text: This looks like a nice trick. Anyways, it looks like it makes your regressed model "outlier independent" . I read the paper very quickly, hence I might be misunderstanding the majority of their proposal. I will definitely try this trick! Tyvm
Answers.UserDisplayName: MttRch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-25T13:08:31.310Z
Answers.LastActivityDate: 2022-02-25T13:08:31.310Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I would like to train a NN in order to approximate an unknown function <span class="math-container">$y = f(x_1,x_2)$</span>. I have a lot of measurements <span class="math-container">$y = [y_1,\dots,y_K]$</span> (with K that could be in the range of 10-100 thousands) which come out from either a simulation or a measurement of a system. I've built a feed-forward NN for solving this problem by using a MSE loss-function, i.e.</p>
  <p><span class="math-container">$$\mathcal{L} = \frac{1}{K}\sum_{i=1}^K(y_i-\hat{y}_i)^2$$</span></p>
  <p>where I defined as <span class="math-container">$\hat{y}$</span> the prediction of the NN. As per activation function I used a <em>ReLU</em>. The network topology is fairly simple with <em>input layer</em> having two neurons <span class="math-container">$(x_1,x_2)$</span>, three <em>hidden layers</em> with 10 neurons each and finally the <em>output layer</em> (single output neuron).</p>
  <p>After the training process expires I've obtained a very curious result. The loss function assumes very small values hence (apparently) indicating that the training is successful. However, if I analyse the squared-error point-by-point, i.e. the quantity</p>
  <p><span class="math-container">$$ \boldsymbol{\epsilon}_y = [(y_1-\hat{y}_1)^2,\dots,(y_K-\hat{y}_K)^2] $$</span>
  I find that for the vast majority of points said quantity is basically zero, with exception of some &quot;outliers&quot; where the error is huge. It looks like this event happens where the gradient of <span class="math-container">$f()$</span> is rather big, which might be a reasonable assumption maybe.</p>
  <p>I would like for this to not happen anymore. I'd rather accept a slightly bigger error throughout the whole function domain then have the majority of points with null error but some local points that are completely off. As a requirement, the network topology shall be kept rather &quot;easy&quot; so I would not like to increase the number of layers and/or neurons per layer. As a side node, I've also tried to increase the topology complexity a little bit (i.e. 15 neurons per hidden layer and adding a 4th hidden layer) obtaining slighlty better results but still unacceptable error around the steepest points of the function.</p>
  <p>I've got two ideas for now:</p>
  <ol>
  <li>Use a different loss-function <span class="math-container">$\mathcal{L}$</span></li>
  <li>Sample the dataset <span class="math-container">$(x_1,x_2)$</span> more frequently around steepest regions, and less frequently where the function is rather smooth and flat</li>
  </ol>
  <h3>First option</h3>
  <p>A different loss-function to be adopted. I'm not very familiar with loss-function that might help solve my problem, some rather quick research yielded no good results and I found no significant literature highlighting this kind of problem. I initially thought of something of the likes of</p>
  <p><span class="math-container">$$ \mathcal{L} = \frac{1}{K}\sum_{i=1}^K(y_i-\hat{y}_i)^2 + \mu\, max\{\boldsymbol{\epsilon}_y\} $$</span>
  where <span class="math-container">$\mu$</span> is some tuned parameter that penalizes the maximum error. I'm not sure if this makes sense and if it increases the training complexity to the extent that the training process runs in no convergence territory.</p>
  <h3>Second option</h3>
  <p>I am not sure how to formalize the undersampling where the function is smooth and flat, I imagine that some kernel used for image processing (edge detection kind of kernels of some sort?) might be helpful but I'm in completely unknown territory.</p>
  <h3>Conclusion</h3>
  <p>I'm looking for some insight and ideas (literature results or analogous cases linked are a huge plus!) for solving this curious problems.</p>
  <p>Thanks for the help!</p>
  <h3>Edit</h3>
  <p>I realize that without having graphical indsights of what I'm doing, it's going to be rather difficult to have a clear understanding. The function <span class="math-container">$f(x_1,x_2)$</span> is plotted here below (let alone the red line, it is just there to highlight what happens when <span class="math-container">$x_1=\text{M}=1.2$</span> is fixed to a certain value). The missing points of around the variable <span class="math-container">$x_2=f_n=1$</span> represent a singularity in my model and I don't care about them being predicted since they physically have no pratical use. I removed them completely for this reason.</p>
  <p><a href="https://i.sstatic.net/sgdS1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sgdS1.png" alt="" /></a></p>
  <p>Once the training of my NN is complete, I plot the <span class="math-container">$L2$</span> error (MSE) between predicted and true value
  <a href="https://i.sstatic.net/Iv2do.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Iv2do.png" alt="" /></a>
  As you can see, it looks like the points where there is a significant gradient change cannot be predicted correctly.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-21T20:23:25.593Z
Comments.Text: Can you please put your **specific question** in the title? "Neural network for regression problem: inaccuracies" is not specific and it's also not a question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-22T07:50:36.017Z
Comments.Text: @nbro I changed the title, maybe it is a little more specific. It is difficult to put in few words
Comments.UserDisplayName: MttRch
Tags: <neural-networks><optimization><regression><logistic-regression>
Title: Why does my regression-NN completely fail to predict some points?
UserDisplayName: MttRch
ThreadText: # Why does my regression-NN completely fail to predict some points?
  
  (Asked by: username_0 on 2022-02-20)
  
  I would like to train a NN in order to approximate an unknown function $y = f(x\_1,x\_2)$. I have a lot of measurements $y = [y\_1,\dots,y\_K]$ (with K that could be in the range of 10-100 thousands) which come out from either a simulation or a measurement of a system. I've built a feed-forward NN for solving this problem by using a MSE loss-function, i.e.
  
  $$\mathcal{L} = \frac{1}{K}\sum\_{i=1}^K(y\_i-\hat{y}\_i)^2$$
  
  where I defined as $\hat{y}$ the prediction of the NN. As per activation function I used a *ReLU*. The network topology is fairly simple with *input layer* having two neurons $(x\_1,x\_2)$, three *hidden layers* with 10 neurons each and finally the *output layer* (single output neuron).
  
  After the training process expires I've obtained a very curious result. The loss function assumes very small values hence (apparently) indicating that the training is successful. However, if I analyse the squared-error point-by-point, i.e. the quantity
  
  $$ \boldsymbol{\epsilon}\_y = [(y\_1-\hat{y}\_1)^2,\dots,(y\_K-\hat{y}\_K)^2] $$
  I find that for the vast majority of points said quantity is basically zero, with exception of some "outliers" where the error is huge. It looks like this event happens where the gradient of $f()$ is rather big, which might be a reasonable assumption maybe.
  
  I would like for this to not happen anymore. I'd rather accept a slightly bigger error throughout the whole function domain then have the majority of points with null error but some local points that are completely off. As a requirement, the network topology shall be kept rather "easy" so I would not like to increase the number of layers and/or neurons per layer. As a side node, I've also tried to increase the topology complexity a little bit (i.e. 15 neurons per hidden layer and adding a 4th hidden layer) obtaining slighlty better results but still unacceptable error around the steepest points of the function.
  
  I've got two ideas for now:
  
  1. Use a different loss-function $\mathcal{L}$
  2. Sample the dataset $(x\_1,x\_2)$ more frequently around steepest regions, and less frequently where the function is rather smooth and flat
  
  ### First option
  
  A different loss-function to be adopted. I'm not very familiar with loss-function that might help solve my problem, some rather quick research yielded no good results and I found no significant literature highlighting this kind of problem. I initially thought of something of the likes of
  
  $$ \mathcal{L} = \frac{1}{K}\sum\_{i=1}^K(y\_i-\hat{y}\_i)^2 + \mu\, max\{\boldsymbol{\epsilon}\_y\} $$
  where $\mu$ is some tuned parameter that penalizes the maximum error. I'm not sure if this makes sense and if it increases the training complexity to the extent that the training process runs in no convergence territory.
  
  ### Second option
  
  I am not sure how to formalize the undersampling where the function is smooth and flat, I imagine that some kernel used for image processing (edge detection kind of kernels of some sort?) might be helpful but I'm in completely unknown territory.
  
  ### Conclusion
  
  I'm looking for some insight and ideas (literature results or analogous cases linked are a huge plus!) for solving this curious problems.
  
  Thanks for the help!
  
  ### Edit
  
  I realize that without having graphical indsights of what I'm doing, it's going to be rather difficult to have a clear understanding. The function $f(x\_1,x\_2)$ is plotted here below (let alone the red line, it is just there to highlight what happens when $x\_1=\text{M}=1.2$ is fixed to a certain value). The missing points of around the variable $x\_2=f\_n=1$ represent a singularity in my model and I don't care about them being predicted since they physically have no pratical use. I removed them completely for this reason.
  
  [![](https://i.sstatic.net/sgdS1.png)](https://i.sstatic.net/sgdS1.png)
  
  Once the training of my NN is complete, I plot the $L2$ error (MSE) between predicted and true value
  [![](https://i.sstatic.net/Iv2do.png)](https://i.sstatic.net/Iv2do.png)
  As you can see, it looks like the points where there is a significant gradient change cannot be predicted correctly.
  
  **Tags:** neural-networks, optimization, regression, logistic-regression
  
  > **Comments:**
  > **username_1**: Can you please put your **specific question** in the title? "Neural network for regression problem: inaccuracies" is not specific and it's also not a question.  
  > **username_0**: @username_1 I changed the title, maybe it is a little more specific. It is difficult to put in few words  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-25. Score: 1
  
  You could try to replace MSE with the recently proposed [General and Adaptive Robust Loss Function](https://arxiv.org/pdf/1701.03077.pdf), a generalization of several losses among which l2 norm (which aside from a square root is equal to MSE). The neat property of this loss is that it uses a dynamic parameter $\alpha$ which makes the loss behaving differently, that can be learned during training. In this way the loss itself adapt to specific outliers in your data (the authors made a [video](https://www.youtube.com/watch?v=BmNKbnF69eY) with great visuals to explain this point).
  
  I suspect at first glance (i.e. without data examples) that the problem your dealing with is that steep points are treated by your network as outliers, i.e. your model learn to generalize a smooth function cause the function is smooth most of the time, and surely also cause the amount of parameters of your network is really small (so it has a hard time fitting local irregular patterns). If it true that your function is smooth most of the time than oversampling steep points would be a mistake, cause that would mean training your network on a non representative sample of points, hence if this adaptive loss doesn't improve much your results I would just make the network larger.
  
  > **Comments:**
  > **username_0**: This looks like a nice trick. Anyways, it looks like it makes your regressed model "outlier independent" . I read the paper very quickly, hence I might be misunderstanding the majority of their proposal. I will definitely try this trick! Tyvm  
  
  ---
  
  