Answers.Body: <p>The problem was due to the following issues in my implementation:</p>
  <ul>
  <li>The offspring generated in the crossover was not mutated (!)</li>
  <li>The mutations did not occur with the expected frequencies (too few links and weight mutations)</li>
  <li>The sigmoid activation had to be steepened</li>
  </ul>
  <p>Another thing that previously caused issues was the network.activate function. Make sure that you wait for the network to stabilize when doing classification tasks, so all signals have time to propagate through the network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-02T15:00:25.173Z
Answers.LastActivityDate: 2020-11-02T15:00:25.173Z
Answers.UserDisplayName: Sebastian Allard
Body: <p>My implementation of NEAT consistently fails to solve XOR completely. The species converge on different sub-optimal networks which map all input examples but one correctly (most commonly (1,1,0)). Do you have any ideas as to why that is?</p>
  <p>Some information which might be relevant:</p>
  <ul>
  <li>I use a plain logistic activation function in each non-input node 1/(1 + exp(-x)).</li>
  <li>Some of the weights seem to grow quite large in magnitude after a large number of epochs.</li>
  <li>I use the sum squared error as the fitness function.</li>
  <li>Anything over 0.5 is considered a 1 (for comparing the output with the expected)</li>
  </ul>
  <p>Here is one example of an evolved network. Node 0 is a bias node, the other red node is the output, the green are inputs and the blue &quot;hidden&quot;. Disregard the labels on the connections.
  <a href="https://i.sstatic.net/VaZcM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VaZcM.png" alt="enter image description here" /></a></p>
  <p>EDIT: following the XOR suggestions on the <a href="https://www.cs.ucf.edu/%7Ekstanley/neat.html" rel="nofollow noreferrer">NEAT users page</a> of steepening the gain of the sigmoid function, a network that solved XOR was found for the first time after ca 50 epochs. But it still fails most of the time. Here is the network which successfully solved XOR:
  <a href="https://i.sstatic.net/8bEmj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8bEmj.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><activation-functions><neat><neuroevolution>
Title: Evolved networks fail to solve XOR
UserDisplayName: Sebastian Allard
ThreadText: # Evolved networks fail to solve XOR
  
  (Asked by: username_0 on 2020-10-29)
  
  My implementation of NEAT consistently fails to solve XOR completely. The species converge on different sub-optimal networks which map all input examples but one correctly (most commonly (1,1,0)). Do you have any ideas as to why that is?
  
  Some information which might be relevant:
  
  * I use a plain logistic activation function in each non-input node 1/(1 + exp(-x)).
  * Some of the weights seem to grow quite large in magnitude after a large number of epochs.
  * I use the sum squared error as the fitness function.
  * Anything over 0.5 is considered a 1 (for comparing the output with the expected)
  
  Here is one example of an evolved network. Node 0 is a bias node, the other red node is the output, the green are inputs and the blue "hidden". Disregard the labels on the connections.
  [![enter image description here](https://i.sstatic.net/VaZcM.png)](https://i.sstatic.net/VaZcM.png)
  
  EDIT: following the XOR suggestions on the [NEAT users page](https://www.cs.ucf.edu/%7Ekstanley/neat.html) of steepening the gain of the sigmoid function, a network that solved XOR was found for the first time after ca 50 epochs. But it still fails most of the time. Here is the network which successfully solved XOR:
  [![enter image description here](https://i.sstatic.net/8bEmj.png)](https://i.sstatic.net/8bEmj.png)
  
  **Tags:** neural-networks, activation-functions, neat, neuroevolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-11-02. Score: 1 | Accepted answer
  
  The problem was due to the following issues in my implementation:
  
  * The offspring generated in the crossover was not mutated (!)
  * The mutations did not occur with the expected frequencies (too few links and weight mutations)
  * The sigmoid activation had to be steepened
  
  Another thing that previously caused issues was the network.activate function. Make sure that you wait for the network to stabilize when doing classification tasks, so all signals have time to propagate through the network.
  
  
  ---
  
  