Answers.Body: <h1>Yes, definitely.</h1>
  <p>In the simplest example, predicting an output value for a time series <em>is</em> classification. You take in the previous time steps and <em>classify</em> what is the most likely next value. You could do this with a <strong>RNN</strong> (<strong>R</strong>ecurrent <strong>N</strong>eural <strong>N</strong>etwork) for example.</p>
  <p>If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in <span class="math-container">$n$</span> dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.</p>
  <p>As a concrete example, try and approximate the function <span class="math-container">$y = x^3 + x^2 -x -1$</span> by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a <strong>ReLU</strong> (<strong>Re</strong>ctified <strong>L</strong>inear <strong>U</strong>nit) you can quite easily approximate this function. <a href="https://www.desmos.com/calculator/cfvtjusqmq" rel="nofollow noreferrer">See this implementation on desmos</a>.</p>
  <p>If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T04:24:06.593Z
Answers.Text: Thanks for you answer. I would just like to check if I'm getting this right, would the desmos implementation represent a neural network with 1 input (value of x), 6 hidden neurons with ReLU activation and 1 output neuron (predicted value of the funciton), where the output of the output neuron is just the sum of hidden neuron outputs?
Answers.UserDisplayName: user1477107
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T05:58:49.217Z
Answers.Text: Yep that's exactly it. And you can imagine the values inside the functions as weights and bias'. For example, for $n_1$ the weight would be -5 and the bias -7.7. Then from the hidden layer to the final layer these weights are either -1 or +1 as seen by the mix of positive and negative in the calculation of $o(x)$
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T07:29:21.380Z
Answers.Text: I don't know if this belongs to a comment, but would it be possible to show the importance of the bias neuron in a similar fashion (on desmos)?
Answers.UserDisplayName: user1477107
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T03:06:42.467Z
Answers.Text: @user1477107 To see the importance, try matching $y=x^2$ using ReLU, but don't use a bias (as in, you can only multiply x by a value, but can never add to it). You'll notice pretty quick that the issue here is that you are stuck with a max(n, x) function, though perhaps scaled differently. You can never reach any sort of complexity. If you try again, but you are able to add a bias, you can align and move around the functions so that they actually match $y=x^2$. Essentially, a bias lets you *translate* the function
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T04:08:15.607Z
Answers.LastActivityDate: 2020-09-24T04:08:15.607Z
Answers.UserDisplayName: Recessive
Body: <p>While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable.
  So I wonder, if we have a neural network whose task is something else, say predicting an output value of a time series, is it still important to have an activation function that is non linear?</p>
  
Tags: <neural-networks><classification><activation-functions><time-series>
Title: Do we need non-linear activation function in neural networks whose task isn't classification?
UserDisplayName: user1477107
ThreadText: # Do we need non-linear activation function in neural networks whose task isn't classification?
  
  (Asked by: username_0 on 2020-09-24)
  
  While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable.
  So I wonder, if we have a neural network whose task is something else, say predicting an output value of a time series, is it still important to have an activation function that is non linear?
  
  **Tags:** neural-networks, classification, activation-functions, time-series
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-24. Score: 2
  
  # Yes, definitely.
  
  In the simplest example, predicting an output value for a time series *is* classification. You take in the previous time steps and *classify* what is the most likely next value. You could do this with a **RNN** (**R**ecurrent **N**eural **N**etwork) for example.
  
  If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in $n$ dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.
  
  As a concrete example, try and approximate the function $y = x^3 + x^2 -x -1$ by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a **ReLU** (**Re**ctified **L**inear **U**nit) you can quite easily approximate this function. [See this implementation on desmos](https://www.desmos.com/calculator/cfvtjusqmq).
  
  If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.
  
  > **Comments:**
  > **username_0**: Thanks for you answer. I would just like to check if I'm getting this right, would the desmos implementation represent a neural network with 1 input (value of x), 6 hidden neurons with ReLU activation and 1 output neuron (predicted value of the funciton), where the output of the output neuron is just the sum of hidden neuron outputs?  
  > **username_1**: Yep that's exactly it. And you can imagine the values inside the functions as weights and bias'. For example, for $n_1$ the weight would be -5 and the bias -7.7. Then from the hidden layer to the final layer these weights are either -1 or +1 as seen by the mix of positive and negative in the calculation of $o(x)$  
  > **username_0**: I don't know if this belongs to a comment, but would it be possible to show the importance of the bias neuron in a similar fashion (on desmos)?  
  > **username_1**: @username_0 To see the importance, try matching $y=x^2$ using ReLU, but don't use a bias (as in, you can only multiply x by a value, but can never add to it). You'll notice pretty quick that the issue here is that you are stuck with a max(n, x) function, though perhaps scaled differently. You can never reach any sort of complexity. If you try again, but you are able to add a bias, you can align and move around the functions so that they actually match $y=x^2$. Essentially, a bias lets you *translate* the function  
  
  ---
  
  