Answers.Body: <p>Neural network is basically a composition of matrix multiplications (linear combination) and non-linear activation. In classical libraries, when an activation function is not specified then everything stays linear (with the linear activation function y=x).</p>
  <p>These non-linear activation functions have several properties : hyperbolic tangent and sigmoid are double saturated, often used in classification task or combined with Normalization layers.</p>
  <p>Cost function has nothing to do with non-linearity, it is just a way to measure how far we currently are from the goal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-07T11:02:32.413Z
Answers.LastActivityDate: 2022-09-07T11:02:32.413Z
Answers.UserDisplayName: nsaura
Body: <p>I have made a neural network from scratch (in java), which is refusing to switch out of linear regression. I have pushed up the layer sizes (it now has 2 hidden layers, both with 5 neurons), and yet when given harsh sloping polynomials to train on, it still predicts values that follow a gradient, <strong>even though this returns high cost</strong>.</p>
  <p>The network is working in that the predictions do somewhat follow the polynomial as best as a line could, but why wont it actually give me predictions that follow a polynomial like the one it trains on?</p>
  <p>I have checked all aspects of training, SGD is working as it should, as is the cost function (MSE), and yet the network just isn't able find a way to minimise cost, it can't seem to break free of linear regression.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-03T09:26:04.690Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-03T13:24:26.157Z
Comments.Text: What kind of activation fuctions do you use?
Comments.UserDisplayName: Veysel Ersoy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-04T07:29:12.157Z
Comments.Text: Ah, currently I don't use any. And now I have just found a resource going through why I need to: https://stackoverflow.com/questions/51442459/why-is-relu-used-in-regression-with-neural-networks. Thank you.
Comments.UserDisplayName: Gamaray
Tags: <regression><linear-regression><mean-squared-error><java>
Title: Multi-layer network only predicts linear trends
UserDisplayName: Gamaray
ThreadText: # Multi-layer network only predicts linear trends
  
  (Asked by: username_0 on 2022-09-03)
  
  I have made a neural network from scratch (in java), which is refusing to switch out of linear regression. I have pushed up the layer sizes (it now has 2 hidden layers, both with 5 neurons), and yet when given harsh sloping polynomials to train on, it still predicts values that follow a gradient, **even though this returns high cost**.
  
  The network is working in that the predictions do somewhat follow the polynomial as best as a line could, but why wont it actually give me predictions that follow a polynomial like the one it trains on?
  
  I have checked all aspects of training, SGD is working as it should, as is the cost function (MSE), and yet the network just isn't able find a way to minimise cost, it can't seem to break free of linear regression.
  
  **Tags:** regression, linear-regression, mean-squared-error, java
  
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_2**: What kind of activation fuctions do you use?  
  > **username_0**: Ah, currently I don't use any. And now I have just found a resource going through why I need to: https://stackoverflow.com/questions/51442459/why-is-relu-used-in-regression-with-neural-networks. Thank you.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-09-07. Score: 0 | Accepted answer
  
  Neural network is basically a composition of matrix multiplications (linear combination) and non-linear activation. In classical libraries, when an activation function is not specified then everything stays linear (with the linear activation function y=x).
  
  These non-linear activation functions have several properties : hyperbolic tangent and sigmoid are double saturated, often used in classification task or combined with Normalization layers.
  
  Cost function has nothing to do with non-linearity, it is just a way to measure how far we currently are from the goal.
  
  
  ---
  
  