Answers.Body: <p>A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network.</p>
  <p>If we had multiple linearly activated layers, each feeding into each other, the neurons in the previous layer would calculate some weighted sum of the input and send it to the next layer as input, where the next layer's neurons would also calculate a weighted sum on that input, and, in turn, it fires based on another linear activation function.</p>
  <p>No matter how many layers and neurons there are, if all are linear in nature, the final activation function of the last layer is also a linear function of the input of the first layer.</p>
  <p>That means that any or all of these layers can be replaced by one layer. This completely loses the advantage of stacking layers because any multilayer network is equivalent to a single layer with linear activation, given that a combination of linear functions in a linear manner is still another linear function.</p>
  <p>A good way to visualize this messing around with <a href="http://playground.tensorflow.org/#activation=linear&amp;regularization=L2&amp;batchSize=30&amp;dataset=circle&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0.003&amp;noise=0&amp;networkShape=8,8,8,8,8,8&amp;seed=0.18289&amp;showTestData=false&amp;discretize=true&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=true&amp;cosY=false&amp;sinY=true&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false" rel="nofollow noreferrer">Tensorflow playground</a>, which has a spiral data set similar to your data.</p>
  <p>In contrast to the failure of linear activation functions for this data in the previous link, check out <a href="http://playground.tensorflow.org/#activation=tanh&amp;regularization=L1&amp;batchSize=10&amp;dataset=spiral&amp;regDataset=reg-plane&amp;learningRate=0.01&amp;regularizationRate=0&amp;noise=10&amp;networkShape=8,7,2&amp;seed=0.04414&amp;showTestData=false&amp;discretize=true&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=true&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=true&amp;cosY=false&amp;sinY=true&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false" rel="nofollow noreferrer">this much smaller network using a Tanh activation</a> function which can separate the data within 100 or so iterations.</p>
  <p>For further reference on activation functions, checkout <a href="https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0" rel="nofollow noreferrer">this blog post</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-18T21:48:14.930Z
Answers.Text: how does this answer fit with the [Universal Approximation Theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem)? The OP's space in bounded, and the curve is continuous, so a single-hidden-layer network of sufficient width should be capable of learning the curve even with linear activation.
Answers.UserDisplayName: beldaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-01T20:20:12.747Z
Answers.Text: The condition on the activation function $\varphi$ for the UAT to hold is that $\varphi$ is bounded, continuous and nonconstant. With linear $\varphi$ it's either not bounded or a constant--it doesn't matter that the space is bounded.
Answers.UserDisplayName: Chris Cundy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-08-04T00:46:50.980Z
Answers.LastActivityDate: 2021-05-21T19:13:59.313Z
Answers.LastEditDate: 2021-05-21T19:13:59.313Z
Answers.UserDisplayName: Jaden Travnik
Answers.Body: <p>Since I can't comment, there are a few caveats to previous answers.</p>
  
  <p>For instance, if you knew beforehand what the expected boundary function for that variable was, then you could transform it first. 
  For instance, if you knew one feature was expected to be sinusoidal, you could transform your data (theta) using <span class="math-container">$f(x) = a*sin(\theta)$</span> first then expect the variable to be linear. At that point it would fit a linear function and solve for the amplitude instead. Of course, in large problems, this may be impractical, but still possible.</p>
  
  <p>Likewise with quadratic data. I could add 1 new feature for <strong>every</strong> continuous variable that looks like <span class="math-container">$x^2$</span> and capture any quadratic dependence. </p>
  
  <p>Another solution in your shown example would be if you <strong>knew</strong> the data was piece-wise, you could train 3 small, easily trained, NNs for instance with different boundaries. During each new training, you set new boundaries for your functions and see which produces the best results. In your case, this would be something close to <span class="math-container">$[-\infty,-0.5], [-0.5,0.5]$</span>, and <span class="math-container">$[0.5,\infty]$</span>.
  You could argue that this is a single piece-wise neural net just like any traditional piece-wise function. 
  While this last approach seems very simplistic/impractical, this is actually done in practice. For instance in autonomous driving, a separate NN could be trained for object detection (human, lights, signs etc.), another for steering wheel positioning for different speeds,conditions, and turning radius, another for how to accelerate, etc. and then the system runs them all and combines their outputs into a full driving car. Each output is then its boundary of the puzzle on what region of "driving space" it handles.</p>
  
  <p>So while I would say the simple answer is no, there are still ways around it if you have some prior knowledge.</p>
  
  <p>As a response for the Universal Approximation Theorem, see this <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">wiki</a>, which says</p>
  
  <blockquote>
    <p>One of the first versions of the theorem was proved by George Cybenko in 1989 for sigmoid activation functions. It was later shown that the class of deep neural networks is a universal approximator if and only if the activation function is not polynomial."</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-23T17:17:51.400Z
Answers.LastActivityDate: 2020-04-01T17:23:51.580Z
Answers.LastEditDate: 2020-04-01T17:23:51.580Z
Answers.UserDisplayName: brian_ds
Body: <p>From a lecture in machine learning, I know that a linear activation function can only produce a linear function, but I don't know if it can produce a connected linear function, like the one in the image below. This function consists of multiple concatenated lines.</p>
  <p><a href="https://i.sstatic.net/y51Kh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/y51Kh.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-03T09:21:58.877Z
Comments.Text: I don't think so... that would make the output non-linear
Comments.UserDisplayName: Thomas Wagenaar
Tags: <neural-networks><machine-learning><function-approximation>
Title: Can a neural network with linear activation functions produce a connection of linear functions?
UserDisplayName: HansPeterLoft
ThreadText: # Can a neural network with linear activation functions produce a connection of linear functions?
  
  (Asked by: username_0 on 2017-08-03)
  
  From a lecture in machine learning, I know that a linear activation function can only produce a linear function, but I don't know if it can produce a connected linear function, like the one in the image below. This function consists of multiple concatenated lines.
  
  [![enter image description here](https://i.sstatic.net/y51Kh.png)](https://i.sstatic.net/y51Kh.png)
  
  **Tags:** neural-networks, machine-learning, function-approximation
  > **Comments:**
  > **username_1**: I don't think so... that would make the output non-linear  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-08-04. Score: 9
  
  A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network.
  
  If we had multiple linearly activated layers, each feeding into each other, the neurons in the previous layer would calculate some weighted sum of the input and send it to the next layer as input, where the next layer's neurons would also calculate a weighted sum on that input, and, in turn, it fires based on another linear activation function.
  
  No matter how many layers and neurons there are, if all are linear in nature, the final activation function of the last layer is also a linear function of the input of the first layer.
  
  That means that any or all of these layers can be replaced by one layer. This completely loses the advantage of stacking layers because any multilayer network is equivalent to a single layer with linear activation, given that a combination of linear functions in a linear manner is still another linear function.
  
  A good way to visualize this messing around with [Tensorflow playground](http://playground.tensorflow.org/#activation=linear®ularization=L2&batchSize=30&dataset=circle®Dataset=reg-plane&learningRate=0.03®ularizationRate=0.003&noise=0&networkShape=8,8,8,8,8,8&seed=0.18289&showTestData=false&discretize=true&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false), which has a spiral data set similar to your data.
  
  In contrast to the failure of linear activation functions for this data in the previous link, check out [this much smaller network using a Tanh activation](http://playground.tensorflow.org/#activation=tanh®ularization=L1&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.01®ularizationRate=0&noise=10&networkShape=8,7,2&seed=0.04414&showTestData=false&discretize=true&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=false&ySquared=false&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false) function which can separate the data within 100 or so iterations.
  
  For further reference on activation functions, checkout [this blog post](https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0).
  
  > **Comments:**
  > **username_3**: how does this answer fit with the [Universal Approximation Theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem)? The OP's space in bounded, and the curve is continuous, so a single-hidden-layer network of sufficient width should be capable of learning the curve even with linear activation.  
  > **username_4**: The condition on the activation function $\varphi$ for the UAT to hold is that $\varphi$ is bounded, continuous and nonconstant. With linear $\varphi$ it's either not bounded or a constant--it doesn't matter that the space is bounded.  
  
  ---
  
  ### Answer by username_5 on 2020-03-23. Score: 2
  
  Since I can't comment, there are a few caveats to previous answers.
  
  For instance, if you knew beforehand what the expected boundary function for that variable was, then you could transform it first.
  For instance, if you knew one feature was expected to be sinusoidal, you could transform your data (theta) using $f(x) = a\*sin(\theta)$ first then expect the variable to be linear. At that point it would fit a linear function and solve for the amplitude instead. Of course, in large problems, this may be impractical, but still possible.
  
  Likewise with quadratic data. I could add 1 new feature for **every** continuous variable that looks like $x^2$ and capture any quadratic dependence.
  
  Another solution in your shown example would be if you **knew** the data was piece-wise, you could train 3 small, easily trained, NNs for instance with different boundaries. During each new training, you set new boundaries for your functions and see which produces the best results. In your case, this would be something close to $[-\infty,-0.5], [-0.5,0.5]$, and $[0.5,\infty]$.
  You could argue that this is a single piece-wise neural net just like any traditional piece-wise function.
  While this last approach seems very simplistic/impractical, this is actually done in practice. For instance in autonomous driving, a separate NN could be trained for object detection (human, lights, signs etc.), another for steering wheel positioning for different speeds,conditions, and turning radius, another for how to accelerate, etc. and then the system runs them all and combines their outputs into a full driving car. Each output is then its boundary of the puzzle on what region of "driving space" it handles.
  
  So while I would say the simple answer is no, there are still ways around it if you have some prior knowledge.
  
  As a response for the Universal Approximation Theorem, see this [wiki](https://en.wikipedia.org/wiki/Universal_approximation_theorem), which says
  
  > One of the first versions of the theorem was proved by George Cybenko in 1989 for sigmoid activation functions. It was later shown that the class of deep neural networks is a universal approximator if and only if the activation function is not polynomial."
  
  
  ---
  
  