Answers.Body: <p>2 perceptrons without bias (+1 in the output layer, to get the result as 1 number).</p>
  
  <p><a href="https://i.sstatic.net/xp9NG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xp9NG.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-25T18:23:08.257Z
Answers.Text: Welcome to AI!  Love the graphic, but feel free to elaborate a little, and share your insights.
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T03:56:16.313Z
Answers.Text: @DukeZhou This is a matlab representation of 2 layer NN. it stats with inputs (green box), the input is a vector 2x1, these inputs are passed to 2 artificial neurons through weights "w", summed up "+" and passed through activation function "bipolar sigmoid". The Layer 2 is an output layer, doing the same, but the output is just linear (in other words, there is no specific activation function because for linear function input=output).
Answers.UserDisplayName: new_stacker
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T03:58:11.430Z
Answers.Text: I recommend to learn the matlab toolkit, it is simple, provides all information (can print all the weights) and you can supervise the learning process. Or, just code it on your own, 2 layer NN is simple and you can check all the values on your own. DONT use biases for this simple case, that is just adding more hyper parameters.
Answers.UserDisplayName: new_stacker
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-25T14:24:09.887Z
Answers.LastActivityDate: 2018-04-25T14:24:09.887Z
Answers.UserDisplayName: new_stacker
Answers.Body: <p>I'd bet, you're doing something wrong, though I can't tell what it is. Try to change the learning rate dynamically, try to train in varying order, ....</p>
  
  <p>On the seconds thought, it looks like you're using the standard sigmoid function. Then you're doing it basically wrong. The input can only be exactly 1 if the input is infinite - or very big so that the floating point arithmetic outputs 1 after rounding.</p>
  
  <p>That's very wrong for two reasons:</p>
  
  <ul>
  <li>You're forcing the network in a broken state having huge weights and tiny derivatives. That feels like imposing numerical instability on an otherwise sane algorithm. Just don't do it. Map your booleans better (see below).</li>
  <li>You're doing what you don't need. Any value close enough to the wanted result (0 or 1) can be simply evaluated as correct. When you get 0.9 instead of 1, then you can simply stop saying "that's perfect". Remember, all you want is an boolean.</li>
  </ul>
  
  <p>A better mapping would be false=0.1 and true=0.9. This doesn't lead to needing infinite weights and reduces related problems.</p>
  
  <p>Even better may be using a symmetrical activation function (e.g., <code>tanh</code>) and a symmetrical mapping like false=-0.9 and true=0.9.</p>
  
  <p>Also consider using <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-28T19:21:02.200Z
Answers.LastActivityDate: 2018-04-28T19:21:02.200Z
Answers.UserDisplayName: maaartinus
Answers.Body: <p>The initialization of the weights has a big impact on the results. I'm not sure specifically for the XOR gate, but the error can have a local minimum that the network can get &quot;stuck&quot; in during training. Using stochastic gradient descent can help give some randomness that gets the error out of these pits. Also, for the sigmoid function, weights should be initialized so that the input to the activation is close to the part with the highest derivative so that training is better.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T13:00:27.423Z
Answers.Text: It looks like you are right: https://datascience.stackexchange.com/a/21792
Answers.UserDisplayName: rdalmeida
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T20:03:26.793Z
Answers.Text: Its good to read some literature on neural networks like this it explains everything: http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
Answers.UserDisplayName: dan dan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-26T03:24:02.123Z
Answers.LastActivityDate: 2021-01-30T19:01:17.380Z
Answers.LastEditDate: 2021-01-30T19:01:17.380Z
Answers.UserDisplayName: dan dan
Body: <p>I'm trying to understand what would be the best neural network for implementing an XOR gate. I'm considering a neural network to be good if it can produce all the expected outcomes with the lowest possible error.</p>
  <p>It looks like my initial choice of random weights has a big impact on my end result after training. The accuracy (i.e. error) of my neural net is varying a lot depending on my initial choice of random weights.</p>
  <p>I'm starting with a 2 x 2 x 1 neural net, with a bias in the input and hidden layers, using the sigmoid activation function, with a learning rate of 0.5. Below my initial setup, with weights chosen randomly:</p>
  <p><a href="https://i.sstatic.net/4tv2V.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4tv2V.jpg" alt="enter image description here" /></a></p>
  <p>The initial performance is bad, as one would expect:</p>
  <pre><code>Input | Output | Expected | Error
  (0,0)   0.8845      0       39.117%
  (1,1)   0.1134      0       0.643%
  (1,0)   0.7057      1       4.3306%
  (0,1)   0.1757      1       33.9735%
  </code></pre>
  <p>Then I proceed to train my network through backpropagation, feeding the XOR training set 100,000 times. After training is complete, my new weights are:</p>
  <p><a href="https://i.sstatic.net/bPMix.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bPMix.jpg" alt="enter image description here" /></a></p>
  <p>And the performance improved to:</p>
  <pre><code>Input | Output | Expected | Error
  (0,0)   0.0103      0       0.0053%
  (1,1)   0.0151      0       0.0114%
  (1,0)   0.9838      1       0.0131%
  (0,1)   0.9899      1       0.0051%
  </code></pre>
  <p><strong>So my questions are:</strong></p>
  <ol>
  <li><p>Has anyone figured out the best weights for a XOR neural network with that configuration (i.e. 2 x 2 x 1 with bias) ?</p>
  </li>
  <li><p>Why my initial choice of random weights make a big difference to my end result? I was lucky on the example above but depending on my initial choice of random weights I get, after training, errors as big as 50%, which is very bad.</p>
  </li>
  <li><p>Am I doing anything wrong or making any wrong assumptions?</p>
  </li>
  </ol>
  <hr />
  <p>So below is an example of weights I cannot train, for some unknown reason. I think I might be doing my backpropagation training incorrectly. I'm not using batches and I'm updating my weights on each data point solved from my training set.</p>
  <p>Weights: <code>((-9.2782, -.4981, -9.4674, 4.4052, 2.8539, 3.395), (1.2108, -7.934, -2.7631))</code></p>
  <p><a href="https://i.sstatic.net/Q3dCO.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Q3dCO.jpg" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><training><backpropagation><xor-problem>
Title: What is the best XOR neural network configuration out there in terms of low error?
UserDisplayName: rdalmeida
ThreadText: # What is the best XOR neural network configuration out there in terms of low error?
  
  (Asked by: username_0 on 2018-04-25)
  
  I'm trying to understand what would be the best neural network for implementing an XOR gate. I'm considering a neural network to be good if it can produce all the expected outcomes with the lowest possible error.
  
  It looks like my initial choice of random weights has a big impact on my end result after training. The accuracy (i.e. error) of my neural net is varying a lot depending on my initial choice of random weights.
  
  I'm starting with a 2 x 2 x 1 neural net, with a bias in the input and hidden layers, using the sigmoid activation function, with a learning rate of 0.5. Below my initial setup, with weights chosen randomly:
  
  [![enter image description here](https://i.sstatic.net/4tv2V.jpg)](https://i.sstatic.net/4tv2V.jpg)
  
  The initial performance is bad, as one would expect:
  
  ```
  Input | Output | Expected | Error
  (0,0)   0.8845      0       39.117%
  (1,1)   0.1134      0       0.643%
  (1,0)   0.7057      1       4.3306%
  (0,1)   0.1757      1       33.9735%
  
  ```
  
  Then I proceed to train my network through backpropagation, feeding the XOR training set 100,000 times. After training is complete, my new weights are:
  
  [![enter image description here](https://i.sstatic.net/bPMix.jpg)](https://i.sstatic.net/bPMix.jpg)
  
  And the performance improved to:
  
  ```
  Input | Output | Expected | Error
  (0,0)   0.0103      0       0.0053%
  (1,1)   0.0151      0       0.0114%
  (1,0)   0.9838      1       0.0131%
  (0,1)   0.9899      1       0.0051%
  
  ```
  
  **So my questions are:**
  
  1. Has anyone figured out the best weights for a XOR neural network with that configuration (i.e. 2 x 2 x 1 with bias) ?
  2. Why my initial choice of random weights make a big difference to my end result? I was lucky on the example above but depending on my initial choice of random weights I get, after training, errors as big as 50%, which is very bad.
  3. Am I doing anything wrong or making any wrong assumptions?
  
  ---
  
  So below is an example of weights I cannot train, for some unknown reason. I think I might be doing my backpropagation training incorrectly. I'm not using batches and I'm updating my weights on each data point solved from my training set.
  
  Weights: `((-9.2782, -.4981, -9.4674, 4.4052, 2.8539, 3.395), (1.2108, -7.934, -2.7631))`
  
  [![enter image description here](https://i.sstatic.net/Q3dCO.jpg)](https://i.sstatic.net/Q3dCO.jpg)
  
  **Tags:** neural-networks, training, backpropagation, xor-problem
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2018-04-26. Score: 3 | Accepted answer
  
  The initialization of the weights has a big impact on the results. I'm not sure specifically for the XOR gate, but the error can have a local minimum that the network can get "stuck" in during training. Using stochastic gradient descent can help give some randomness that gets the error out of these pits. Also, for the sigmoid function, weights should be initialized so that the input to the activation is close to the part with the highest derivative so that training is better.
  
  > **Comments:**
  > **username_0**: It looks like you are right: https://datascience.stackexchange.com/a/21792  
  > **username_4**: Its good to read some literature on neural networks like this it explains everything: http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf  
  
  ---
  
  ### Answer by username_1 on 2018-04-25. Score: 2
  
  2 perceptrons without bias (+1 in the output layer, to get the result as 1 number).
  
  [![enter image description here](https://i.sstatic.net/xp9NG.png)](https://i.sstatic.net/xp9NG.png)
  
  > **Comments:**
  > **username_2**: Welcome to AI!  Love the graphic, but feel free to elaborate a little, and share your insights.  
  > **username_1**: @username_2 This is a matlab representation of 2 layer NN. it stats with inputs (green box), the input is a vector 2x1, these inputs are passed to 2 artificial neurons through weights "w", summed up "+" and passed through activation function "bipolar sigmoid". The Layer 2 is an output layer, doing the same, but the output is just linear (in other words, there is no specific activation function because for linear function input=output).  
  > **username_1**: I recommend to learn the matlab toolkit, it is simple, provides all information (can print all the weights) and you can supervise the learning process. Or, just code it on your own, 2 layer NN is simple and you can check all the values on your own. DONT use biases for this simple case, that is just adding more hyper parameters.  
  
  ---
  
  ### Answer by username_3 on 2018-04-28. Score: 1
  
  I'd bet, you're doing something wrong, though I can't tell what it is. Try to change the learning rate dynamically, try to train in varying order, ....
  
  On the seconds thought, it looks like you're using the standard sigmoid function. Then you're doing it basically wrong. The input can only be exactly 1 if the input is infinite - or very big so that the floating point arithmetic outputs 1 after rounding.
  
  That's very wrong for two reasons:
  
  * You're forcing the network in a broken state having huge weights and tiny derivatives. That feels like imposing numerical instability on an otherwise sane algorithm. Just don't do it. Map your booleans better (see below).
  * You're doing what you don't need. Any value close enough to the wanted result (0 or 1) can be simply evaluated as correct. When you get 0.9 instead of 1, then you can simply stop saying "that's perfect". Remember, all you want is an boolean.
  
  A better mapping would be false=0.1 and true=0.9. This doesn't lead to needing infinite weights and reduces related problems.
  
  Even better may be using a symmetrical activation function (e.g., `tanh`) and a symmetrical mapping like false=-0.9 and true=0.9.
  
  Also consider using [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)).
  
  
  ---
  
  