Answers.Body: <p>Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, <code>exp(xi)/sum(exp(xj) for j in all x)</code>, where <code>xi</code> is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/exp(xi)). </p>
  
  <p>Now lets visualize your dataset, ie plot the points of XOR as a function of X,Y on coordinate space.
  <a href="https://i.sstatic.net/13NUe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/13NUe.png" alt="XOR plot image"></a>, [src: <a href="https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]" rel="nofollow noreferrer">https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]</a></p>
  
  <p>Now, whatever you do, you cannot separate the points with a single <code>wx+b</code> straight line, and after passing it to sigmoid, you just squeeze the values for those, ie, a clipped <code>wx+b</code>.</p>
  
  <p>Now, it is still a linear equation. Now when you add another layer, a hidden one, you can operate again on the 1st output, which if you squeeze between 0 and 1 or use something like relu activation, will produce some non linearity, otherwise it will just be <code>(w2(w1*x + b1)+b2</code>, which again is a linear equation not able to separate the classes 0 and 1.</p>
  
  <p>So after adding a hidden layer and passing it through activation function, you get non linear output, example, with sigmoid you get <code>w2*(1/+exp(w2*x+b1))+b2</code> which has a fairly non linear term of x, and after passing it through sigmoid again, you are able to squeeze the outputs(otherwise they can be higher than 1 or lower than 0,but XOR outputs are binary) after fitting the curve so that both classes are separate.</p>
  
  <p>Now, food for thought, what will happen if you use 2 output neurons for XOR (represent 0 as [1,0] and 1 as [0,1])?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T16:07:16.920Z
Answers.Text: Thank you for the answer and a wonderful question. In my opinion, using 2 output neurons will not be sufficient because each of the output from neurons is still just 'clipped' from a linear function. But the xor data(even though it's representation is by a matrix now) is still non linear, we just increased it's space representation from 1d to 2d. It would be great, if you can reason your question
Answers.UserDisplayName: Hrushi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T13:31:01.047Z
Answers.LastActivityDate: 2020-05-04T13:36:14.007Z
Answers.LastEditDate: 2020-05-04T13:36:14.007Z
Answers.UserDisplayName: SajanGohil
Body: <p>xor is a non-linear dataset. It cannot be solved with any number of perceptron based neural network but when the perceptions are applied the sigmoid activation function, we can solve the xor dataset. </p>
  
  <p>But I came across a source where the following statement is stated as <code>False</code> </p>
  
  <p><code>A two layer (one input layer; one output layer; no hidden layer) neural network can represent the XOR function</code>.</p>
  
  <p>However, I have trained a model with no hidden layers, gives the following result:</p>
  
  <pre><code>[INFO] data=[0 0],ground-truth=0, pred=0.5161, step=1
  [INFO] data=[0 1],ground-truth=1, pred=0.5000, step=1
  [INFO] data=[1 0],ground-truth=1, pred=0.4839, step=0
  [INFO] data=[1 1],ground-truth=0, pred=0.4678, step=0
  </code></pre>
  
  <p>So, in if I apply a softmax classifier, I can separate the xor dataset with a nn without any hidden layer. This makes the statement incorrect. </p>
  
  <p>Is it true that we cannot separate a non linear dataset without any hidden layers in a neural network? If yes, where am I wrong in my reasoning from the training of the nn I have done above</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T13:00:56.490Z
Comments.Text: What do you think softmax does? what change will it make if you replace a sigmoid with a softmax when the output layer is only one neuron?
Comments.UserDisplayName: SajanGohil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T13:06:13.707Z
Comments.Text: I meant placing the softmax after the last layer, but now realize that it would no longer be a nn with no hidden lalyers
Comments.UserDisplayName: Hrushi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T13:07:24.140Z
Comments.Text: last layer has weights, you just replace a sigmoid with a softmax
Comments.UserDisplayName: SajanGohil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T13:14:59.597Z
Comments.Text: But adding a softmax or a sigmoid will make the output of the network nonlinear right (because sigmoid or softmax are nonlinear functions), and the xor is a non linear dataset too. Is there any proof/justification that we cannot  separate
Comments.UserDisplayName: Hrushi
Tags: <neural-networks><non-linear-regression>
Title: solving xor function using a neural network with no hidden layers
UserDisplayName: Hrushi
ThreadText: # solving xor function using a neural network with no hidden layers
  
  (Asked by: username_0 on 2020-05-04)
  
  xor is a non-linear dataset. It cannot be solved with any number of perceptron based neural network but when the perceptions are applied the sigmoid activation function, we can solve the xor dataset. 
  But I came across a source where the following statement is stated as False 
  A two layer (one input layer; one output layer; no hidden layer) neural network can represent the XOR function.
  However, I have trained a model with no hidden layers, gives the following result:
  [INFO] data=[0 0],ground-truth=0, pred=0.5161, step=1
  [INFO] data=[0 1],ground-truth=1, pred=0.5000, step=1
  [INFO] data=[1 0],ground-truth=1, pred=0.4839, step=0
  [INFO] data=[1 1],ground-truth=0, pred=0.4678, step=0
  
  So, in if I apply a softmax classifier, I can separate the xor dataset with a nn without any hidden layer. This makes the statement incorrect. 
  Is it true that we cannot separate a non linear dataset without any hidden layers in a neural network? If yes, where am I wrong in my reasoning from the training of the nn I have done above
  
  **Tags:** neural-networks, non-linear-regression
  > **Comments:**
  > **username_1**: What do you think softmax does? what change will it make if you replace a sigmoid with a softmax when the output layer is only one neuron?  
  > **username_0**: I meant placing the softmax after the last layer, but now realize that it would no longer be a nn with no hidden lalyers  
  > **username_1**: last layer has weights, you just replace a sigmoid with a softmax  
  > **username_0**: But adding a softmax or a sigmoid will make the output of the network nonlinear right (because sigmoid or softmax are nonlinear functions), and the xor is a non linear dataset too. Is there any proof/justification that we cannot  separate  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-04. Score: 3 | Accepted answer
  
  Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, `exp(xi)/sum(exp(xj) for j in all x)`, where `xi` is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/exp(xi)).
  
  Now lets visualize your dataset, ie plot the points of XOR as a function of X,Y on coordinate space.
  [![XOR plot image](https://i.sstatic.net/13NUe.png)](https://i.sstatic.net/13NUe.png), [src: <https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]>
  
  Now, whatever you do, you cannot separate the points with a single `wx+b` straight line, and after passing it to sigmoid, you just squeeze the values for those, ie, a clipped `wx+b`.
  
  Now, it is still a linear equation. Now when you add another layer, a hidden one, you can operate again on the 1st output, which if you squeeze between 0 and 1 or use something like relu activation, will produce some non linearity, otherwise it will just be `(w2(w1*x + b1)+b2`, which again is a linear equation not able to separate the classes 0 and 1.
  
  So after adding a hidden layer and passing it through activation function, you get non linear output, example, with sigmoid you get `w2*(1/+exp(w2*x+b1))+b2` which has a fairly non linear term of x, and after passing it through sigmoid again, you are able to squeeze the outputs(otherwise they can be higher than 1 or lower than 0,but XOR outputs are binary) after fitting the curve so that both classes are separate.
  
  Now, food for thought, what will happen if you use 2 output neurons for XOR (represent 0 as [1,0] and 1 as [0,1])?
  
  > **Comments:**
  > **username_0**: Thank you for the answer and a wonderful question. In my opinion, using 2 output neurons will not be sufficient because each of the output from neurons is still just 'clipped' from a linear function. But the xor data(even though it's representation is by a matrix now) is still non linear, we just increased it's space representation from 1d to 2d. It would be great, if you can reason your question  
  
  ---
  
  