<p>Neural network is basically a composition of matrix multiplications (linear combination) and non-linear activation. In classical libraries, when an activation function is not specified then everything stays linear (with the linear activation function y=x).</p>
<p>These non-linear activation functions have several properties : hyperbolic tangent and sigmoid are double saturated, often used in classification task or combined with Normalization layers.</p>
<p>Cost function has nothing to do with non-linearity, it is just a way to measure how far we currently are from the goal.</p>

CC BY-SA 4.0
2022-09-07T11:02:32.413Z
2022-09-07T11:02:32.413Z
nsaura
<p>I have made a neural network from scratch (in java), which is refusing to switch out of linear regression. I have pushed up the layer sizes (it now has 2 hidden layers, both with 5 neurons), and yet when given harsh sloping polynomials to train on, it still predicts values that follow a gradient, <strong>even though this returns high cost</strong>.</p>
<p>The network is working in that the predictions do somewhat follow the polynomial as best as a line could, but why wont it actually give me predictions that follow a polynomial like the one it trains on?</p>
<p>I have checked all aspects of training, SGD is working as it should, as is the cost function (MSE), and yet the network just isn't able find a way to minimise cost, it can't seem to break free of linear regression.</p>

CC BY-SA 4.0
2022-09-03T09:26:04.690Z
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Community
CC BY-SA 4.0
2022-09-03T13:24:26.157Z
What kind of activation fuctions do you use?
Veysel Ersoy
CC BY-SA 4.0
2022-09-04T07:29:12.157Z
Ah, currently I don't use any. And now I have just found a resource going through why I need to: https://stackoverflow.com/questions/51442459/why-is-relu-used-in-regression-with-neural-networks. Thank you.
Gamaray
<regression><linear-regression><mean-squared-error><java>
Multi-layer network only predicts linear trends
Gamaray
# Multi-layer network only predicts linear trends

(Asked by: username_0 on 2022-09-03)

I have made a neural network from scratch (in java), which is refusing to switch out of linear regression. I have pushed up the layer sizes (it now has 2 hidden layers, both with 5 neurons), and yet when given harsh sloping polynomials to train on, it still predicts values that follow a gradient, **even though this returns high cost**.

The network is working in that the predictions do somewhat follow the polynomial as best as a line could, but why wont it actually give me predictions that follow a polynomial like the one it trains on?

I have checked all aspects of training, SGD is working as it should, as is the cost function (MSE), and yet the network just isn't able find a way to minimise cost, it can't seem to break free of linear regression.

**Tags:** regression, linear-regression, mean-squared-error, java

> **Comments:**
> **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
> **username_2**: What kind of activation fuctions do you use?  
> **username_0**: Ah, currently I don't use any. And now I have just found a resource going through why I need to: https://stackoverflow.com/questions/51442459/why-is-relu-used-in-regression-with-neural-networks. Thank you.  

---

## 1 answers

### Answer by username_3 on 2022-09-07. Score: 0 | Accepted answer

Neural network is basically a composition of matrix multiplications (linear combination) and non-linear activation. In classical libraries, when an activation function is not specified then everything stays linear (with the linear activation function y=x).

These non-linear activation functions have several properties : hyperbolic tangent and sigmoid are double saturated, often used in classification task or combined with Normalization layers.

Cost function has nothing to do with non-linearity, it is just a way to measure how far we currently are from the goal.


---

