Answers.Body: <p>Consider what happens if you intend to train a linear classifier on replicating something trivial as the XOR function. If you program/train the classifier (of arbitrary size) such that it outputs <code>XOR condition is met</code> whenever feature <code>a</code> or feature <code>b</code> are present, then the linear classifier will also (incorrectly) output <code>XOR condition is met</code> whenever both features together are present. That is because linear classifiers simply sum up contributions of all features and work with the total weighted inputs they receive. For our example, that means that when the weighted contribution of either feature is sufficient already to trigger the classifier to output <code>XOR condition is met</code>, then obviously also the summed contributions of both features are sufficient to trigger the same response.</p>
  <p>To get a classifier that is capable of outputting <code>XOR condition is met</code> <em>if and only if</em> the summed contributions of all input features are above a lower threshold <em>and</em> below an upper threshold, commonly non-linearities are introduced. You could of course also try to employ a quadratic function to solve the two-feature problem, but as soon as the number of variables/features increases again, you run into the same problem again, only in higher dimensions. Therefore, the most general approach to solving this problem of learning non-linear functions, like the <code>XOR</code>, is by setting up large models with enough capacity to learn a given task and equipping them with non-linearities. That simplifies training since it allows for using stochastic gradient descent for training the system/classifier, preventing one from having to solve Higher-Degree polynomial equations analytically (which can get computationally quite expensive quite quickly) to solve some task.</p>
  <p>In case you are interested, <a href="https://ieeexplore.ieee.org/document/1194667" rel="nofollow noreferrer">here's one paper</a> analyzing and dealing with the <code>XOR</code> problem (as one concrete instance of a problem where purely linear models fail to solve some task).</p>
  <p><em><strong>EDIT:</strong></em></p>
  <p>You can consider a layer in a network as a function <span class="math-container">$y = f(x)$</span>, where <span class="math-container">$x$</span> is the input to some layer <span class="math-container">$f$</span> and <span class="math-container">$y$</span> is the output of the layer. As you propagate <span class="math-container">$x$</span>, being the network's input, through the network, you get something like <span class="math-container">$y = p(t(h(g(f(x)))))$</span>, where <span class="math-container">$f$</span> is the input layer and <span class="math-container">$p$</span> constitutes the output layer, i.e. a set of weights, by which the input to that respective layer gets multiplied.
  If <span class="math-container">$h$</span>, for example, is some non-linear activation function, like <code>ReLU</code> or <code>sigmoid</code>, then <span class="math-container">$y$</span>, being the network's output, is a non-linear function of input <span class="math-container">$x$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T13:32:38.217Z
Answers.Text: I think that a simple diagram that illustrates that a straight-line cannot solve the XOR problem and that you need a non-linear function to solve would give the intuition. I suppose that there are diagrams on the web that show this issue well.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T15:31:00.337Z
Answers.Text: what types of diagrams are you talking about?
Answers.UserDisplayName: Alankar Shukla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-27T22:36:16.730Z
Answers.Text: @AlankarShukla For example, [these diagrams](https://cdn-images-1.medium.com/max/957/1*Tc8UgR_fjI_h0p3y4H9MwA.png) show that a straight-line cannot solve the XOR problem. By the way, next time, tag me with `@nbro`, otherwise, I may not see your message.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-30T03:46:00.023Z
Answers.Text: @nbro i wanna know what is a non linear data.how it can introduced in neural networks,how the neural network learns the non linear data for better prediction.
Answers.UserDisplayName: Alankar Shukla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-30T03:49:05.733Z
Answers.Text: @nbro i can't get the intuition behind it.i want to know what is a non linear data,how it is introduced in neural networks,and how neural networks learn these data for predicting.
Answers.UserDisplayName: Alankar Shukla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-30T12:13:27.743Z
Answers.Text: @AlankarShukla If you don't know what non-linear data means, then ask another question. That's the correct approach. However, make sure that this answer cannot be easily found on the web.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-30T13:13:53.183Z
Answers.Text: @AlankarShukla, In a nutshell, non-linearly separable data is any kind of data where a corresponding decision boundary or decision-(hyper-)plane must be (geometrically) bent, i.e. be a non-linear function of the input parameters, in order to separate two or more classes from one another.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T05:59:47.873Z
Answers.Text: @Daniel B i want to know  that using non linear activation functions in the hidden layers, the input data is again and again normalise.so this process leads to learn a non linear function in Neural network?
Answers.UserDisplayName: Alankar Shukla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T12:29:49.660Z
Answers.Text: It doesn't have anything to do with normalization.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T13:06:44.700Z
Answers.LastActivityDate: 2021-03-28T17:53:35.830Z
Answers.LastEditDate: 2021-03-28T17:53:35.830Z
Answers.UserDisplayName: Daniel B.
Answers.Body: <p>For a regressor, it can work fine to have an output layer that is linear.</p>
  <p>The composition of two linear functions is also linear, so in a deep neural net, if all layers are linear it can only learn a linear function. As Daniel B explains, XOR is a good example of a function with no useful linear approximation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T16:03:52.067Z
Answers.LastActivityDate: 2021-03-29T16:03:52.067Z
Answers.UserDisplayName: chrishmorris
Body: <p>Why is non-linearity desirable in a neural network?</p>
  <p>I couldn't find satisfactory answers to this question on the web. I typically get answers like &quot;real-world problems require non-linear solutions, which are not trivial. So, we use non-linear activation functions for non-linearity&quot;.</p>
  
Tags: <neural-networks><deep-learning><activation-functions>
Title: Why is non-linearity desirable in a neural network?
UserDisplayName: Alankar Shukla
ThreadText: # Why is non-linearity desirable in a neural network?
  
  (Asked by: username_0 on 2020-06-25)
  
  Why is non-linearity desirable in a neural network?
  
  I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear solutions, which are not trivial. So, we use non-linear activation functions for non-linearity".
  
  **Tags:** neural-networks, deep-learning, activation-functions
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-25. Score: 0 | Accepted answer
  
  Consider what happens if you intend to train a linear classifier on replicating something trivial as the XOR function. If you program/train the classifier (of arbitrary size) such that it outputs `XOR condition is met` whenever feature `a` or feature `b` are present, then the linear classifier will also (incorrectly) output `XOR condition is met` whenever both features together are present. That is because linear classifiers simply sum up contributions of all features and work with the total weighted inputs they receive. For our example, that means that when the weighted contribution of either feature is sufficient already to trigger the classifier to output `XOR condition is met`, then obviously also the summed contributions of both features are sufficient to trigger the same response.
  
  To get a classifier that is capable of outputting `XOR condition is met` *if and only if* the summed contributions of all input features are above a lower threshold *and* below an upper threshold, commonly non-linearities are introduced. You could of course also try to employ a quadratic function to solve the two-feature problem, but as soon as the number of variables/features increases again, you run into the same problem again, only in higher dimensions. Therefore, the most general approach to solving this problem of learning non-linear functions, like the `XOR`, is by setting up large models with enough capacity to learn a given task and equipping them with non-linearities. That simplifies training since it allows for using stochastic gradient descent for training the system/classifier, preventing one from having to solve Higher-Degree polynomial equations analytically (which can get computationally quite expensive quite quickly) to solve some task.
  
  In case you are interested, [here's one paper](https://ieeexplore.ieee.org/document/1194667) analyzing and dealing with the `XOR` problem (as one concrete instance of a problem where purely linear models fail to solve some task).
  
  ***EDIT:***
  
  You can consider a layer in a network as a function $y = f(x)$, where $x$ is the input to some layer $f$ and $y$ is the output of the layer. As you propagate $x$, being the network's input, through the network, you get something like $y = p(t(h(g(f(x)))))$, where $f$ is the input layer and $p$ constitutes the output layer, i.e. a set of weights, by which the input to that respective layer gets multiplied.
  If $h$, for example, is some non-linear activation function, like `ReLU` or `sigmoid`, then $y$, being the network's output, is a non-linear function of input $x$.
  
  > **Comments:**
  > **username_2**: I think that a simple diagram that illustrates that a straight-line cannot solve the XOR problem and that you need a non-linear function to solve would give the intuition. I suppose that there are diagrams on the web that show this issue well.  
  > **username_0**: what types of diagrams are you talking about?  
  > **username_2**: @username_0 For example, [these diagrams](https://cdn-images-1.medium.com/max/957/1*Tc8UgR_fjI_h0p3y4H9MwA.png) show that a straight-line cannot solve the XOR problem. By the way, next time, tag me with `@username_2`, otherwise, I may not see your message.  
  > **username_0**: @username_2 i wanna know what is a non linear data.how it can introduced in neural networks,how the neural network learns the non linear data for better prediction.  
  > **username_0**: @username_2 i can't get the intuition behind it.i want to know what is a non linear data,how it is introduced in neural networks,and how neural networks learn these data for predicting.  
  > **username_2**: @username_0 If you don't know what non-linear data means, then ask another question. That's the correct approach. However, make sure that this answer cannot be easily found on the web.  
  > **username_1**: @username_0, In a nutshell, non-linearly separable data is any kind of data where a corresponding decision boundary or decision-(hyper-)plane must be (geometrically) bent, i.e. be a non-linear function of the input parameters, in order to separate two or more classes from one another.  
  > **username_0**: @username_1i want to know  that using non linear activation functions in the hidden layers, the input data is again and again normalise.so this process leads to learn a non linear function in Neural network?  
  > **username_1**: It doesn't have anything to do with normalization.  
  
  ---
  
  ### Answer by username_3 on 2021-03-29. Score: 0
  
  For a regressor, it can work fine to have an output layer that is linear.
  
  The composition of two linear functions is also linear, so in a deep neural net, if all layers are linear it can only learn a linear function. As username_1explains, XOR is a good example of a function with no useful linear approximation.
  
  
  ---
  
  