Answers.Body: <p>One neuron on its own can only solve linearly separable problems. You need a combination of neurons to solve non-linearly separable problems.</p>
  <p>For the XOR case, you need at least 2 neurons at the first layer, and 1 neuron at the output layer to properly classify it.</p>
  <p>Keep in mind that, sometimes, the 3-neurons network might get stuck in a local minimum as well. You will need some luck in the random initialization of weights. Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-10T12:29:05.713Z
Answers.LastActivityDate: 2021-12-12T08:54:22.300Z
Answers.LastEditDate: 2021-12-12T08:54:22.300Z
Answers.UserDisplayName: MrYellow
Body: <p>I'm trying to create a neural network to simulate an XOR gate.</p>
  <p>Here's my dataset:</p>
  <pre><code>╔════════╦════════╗
  ║ x1, x2 ║ y1, y2 ║
  ╠════════╬════════╣
  ║  0, 0  ║  0, 1  ║
  ║  0, 1  ║  1, 0  ║
  ║  1, 0  ║  1, 0  ║
  ║  1, 1  ║  0, 1  ║
  ╚════════╩════════╝
  </code></pre>
  <p>And my neural network:</p>
  <p><a href="https://i.sstatic.net/fJQyj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fJQyj.png" alt="enter image description here" /></a></p>
  <p>I use logistic loss to get the error between target <span class="math-container">$y_{k}$</span> and output <span class="math-container">$\hat{y}_{k}$</span>:</p>
  <p><span class="math-container">$$ E(y_{k}, \hat{y}_{k}) = - y_{k} \cdot log(\hat{y}_{k}) + (1 - y_{k}) \cdot log(1 - \hat{y}_{k}) $$</span></p>
  <p>And then use the chain rule to update the weights. For example weight <span class="math-container">$w_{3}$</span>'s contribution to the error is:</p>
  <p><span class="math-container">$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \sum_{k=1}^{2} \left(\frac{\partial E(y_{k}, \hat{y}_{k})}{\partial \hat{y}_{k}} \cdot \frac{\partial s_{k}}{\partial c_{1}}\right) \cdot \frac{\partial c_{1}}{\partial w_{3}} $$</span></p>
  <p>Which in developed form is:</p>
  <p><span class="math-container">$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \left( \left( - \frac{y_{1}}{\hat{y}_{1}} + \frac{1 - y_{1}}{1 - \hat{y}_{1}} \right) \cdot \hat{y}_{1} \cdot (1 - \hat{y}_{1}) + \left( - \frac{y_{2}}{\hat{y}_{2}} + \frac{1 - y_{2}}{1 - \hat{y}_{2}} \right) \cdot (- \hat{y}_{2}) \cdot\frac{c_{1}}{c_{1} + c_{2}} \right) \cdot s_{0} $$</span></p>
  <p>My issue is that, after a couple epochs of training on the entire dataset, the network always outputs:
  <span class="math-container">$$ \hat{y}_{1} = \hat{y}_{2} = 0.5 $$</span></p>
  <p>What am I doing wrong?</p>
  
Tags: <neural-networks><implementation><softmax><xor-problem><categorical-crossentropy>
Title: Why does my neural network to solve the XOR problem always output 0.5?
UserDisplayName: user45708
ThreadText: # Why does my neural network to solve the XOR problem always output 0.5?
  
  (Asked by: username_0 on 2021-03-25)
  
  I'm trying to create a neural network to simulate an XOR gate.
  
  Here's my dataset:
  
  ```
  ╔════════╦════════╗
  ║ x1, x2 ║ y1, y2 ║
  ╠════════╬════════╣
  ║  0, 0  ║  0, 1  ║
  ║  0, 1  ║  1, 0  ║
  ║  1, 0  ║  1, 0  ║
  ║  1, 1  ║  0, 1  ║
  ╚════════╩════════╝
  
  ```
  
  And my neural network:
  
  [![enter image description here](https://i.sstatic.net/fJQyj.png)](https://i.sstatic.net/fJQyj.png)
  
  I use logistic loss to get the error between target $y\_{k}$ and output $\hat{y}\_{k}$:
  
  $$ E(y\_{k}, \hat{y}\_{k}) = - y\_{k} \cdot log(\hat{y}\_{k}) + (1 - y\_{k}) \cdot log(1 - \hat{y}\_{k}) $$
  
  And then use the chain rule to update the weights. For example weight $w\_{3}$'s contribution to the error is:
  
  $$ \sum\_{k=1}^{2} \frac{\partial E(y\_{k}, \hat{y}\_{k})}{\partial w\_{3}} = \sum\_{k=1}^{2} \left(\frac{\partial E(y\_{k}, \hat{y}\_{k})}{\partial \hat{y}\_{k}} \cdot \frac{\partial s\_{k}}{\partial c\_{1}}\right) \cdot \frac{\partial c\_{1}}{\partial w\_{3}} $$
  
  Which in developed form is:
  
  $$ \sum\_{k=1}^{2} \frac{\partial E(y\_{k}, \hat{y}\_{k})}{\partial w\_{3}} = \left( \left( - \frac{y\_{1}}{\hat{y}\_{1}} + \frac{1 - y\_{1}}{1 - \hat{y}\_{1}} \right) \cdot \hat{y}\_{1} \cdot (1 - \hat{y}\_{1}) + \left( - \frac{y\_{2}}{\hat{y}\_{2}} + \frac{1 - y\_{2}}{1 - \hat{y}\_{2}} \right) \cdot (- \hat{y}\_{2}) \cdot\frac{c\_{1}}{c\_{1} + c\_{2}} \right) \cdot s\_{0} $$
  
  My issue is that, after a couple epochs of training on the entire dataset, the network always outputs:
  $$ \hat{y}\_{1} = \hat{y}\_{2} = 0.5 $$
  
  What am I doing wrong?
  
  **Tags:** neural-networks, implementation, softmax, xor-problem, categorical-crossentropy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-10. Score: 0 | Accepted answer
  
  One neuron on its own can only solve linearly separable problems. You need a combination of neurons to solve non-linearly separable problems.
  
  For the XOR case, you need at least 2 neurons at the first layer, and 1 neuron at the output layer to properly classify it.
  
  Keep in mind that, sometimes, the 3-neurons network might get stuck in a local minimum as well. You will need some luck in the random initialization of weights. Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.
  
  
  ---
  
  