Answers.Body: <p>Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ones.</p>
  
  <p>Your weights should be both positive and negative.</p>
  
  <p>In addition, once trained, they tend to follow a certain size distribution. It helps if you start with values within that range. This is often called Xavier or Glorot initialisation. Basically, if your number of inputs to a layer is <code>n_inputs</code> and number of outputs is <code>n_outputs</code>, you will have <code>n_inputs * n_outputs</code> weights, and if you initialise them with a random number generator, then you should use a multiplier something like this:</p>
  
  <pre><code>w = (rand() - 0.5) * sqrt(6/(n_inputs + n_outputs))
  </code></pre>
  
  <p>Note that <em>is</em> an addition inside the square root, you don't take the total number of weights.</p>
  
  <p>As you are using a GA, you may want to use some form of clipping or normalisation to prevent mutations from drifting too far away from these useful weights. I'm not entirely sure what would be best, as usually I would use back propagation to train a large network. Potentially a maxnorm regularisation would help - choose a value (maybe even make it adjustable via mutation) that the norm of weight matrix in each layer should not exceed (separately for each layer, or one overall value, up to you), and if any learning step creates a network with too high a norm, scale it down. The L2 norm, <code>sqrt(sum_of_squared_weights)</code> is usual choice for maxnorm regularisation, and this works well with backpropagation-based supervised learning.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-07T08:23:03.840Z
Answers.LastActivityDate: 2018-04-07T08:23:03.840Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST</a> classification. The network will be trained purely with <strong>genetic algorithms</strong> instead of back-propagation.</p>
  <p>Here are the details:</p>
  <ul>
  <li>Population size: 50</li>
  <li>Activation function: sigmoid</li>
  <li>Fixed topology</li>
  <li>XOR: 2 inputs, 1 output. Tested with different numbers of hidden layers/nodes.</li>
  <li>MNIST: <span class="math-container">$28*28=784$</span> inputs for all pixels, will be either ON(1) or OFF(0). 10 outputs to represent digits 0-9</li>
  <li>Initial population will be given random weights between 0 and 1</li>
  <li>10 &quot;Fittest&quot; networks survive each iteration, and performs crossover to reproduce 40 offspring</li>
  <li>For all weights, mutation occurs to add a random value between -1 to 1, with a 5% chance</li>
  </ul>
  <p>With 2 hidden layers of 4 and 3 neurons respectively, XOR managed to achieve 97-99.9% accuracy in around 100 generations. Biases were not used here.</p>
  <p><strong>However</strong>, trying out MNIST revealed a pretty glaring issue - the 784 inputs; a large increase of nodes compared to XOR, multiplied with weights and added up results in HUGE values of 50 to even 100, way beyond the typical domain range of the activation function (sigmoid).</p>
  <p>This just renders all layers' outputs as 1 or 0.99999-something, which breaks the entire network. Also, since this makes all individuals in a population extremely similar to one other, the genetic algorithm seems to have no clue on how to improve. The crossover will produce an offspring almost identical to its parents, and some lucky mutations are simply ignored by the sheer amount of other neurons!</p>
  <p>What can be a viable solution to this?</p>
  <p>It's my first time studying NNs, and this is really challenging.</p>
  
Tags: <neural-networks><genetic-algorithms><data-preprocessing><weights-initialization><weight-normalization>
Title: How to solve the problem of too big activations when using genetic algorithms to train neural networks?
UserDisplayName: Joshua Jang
ThreadText: # How to solve the problem of too big activations when using genetic algorithms to train neural networks?
  
  (Asked by: username_0 on 2018-04-06)
  
  I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and [MNIST](http://yann.lecun.com/exdb/mnist/) classification. The network will be trained purely with **genetic algorithms** instead of back-propagation.
  
  Here are the details:
  
  * Population size: 50
  * Activation function: sigmoid
  * Fixed topology
  * XOR: 2 inputs, 1 output. Tested with different numbers of hidden layers/nodes.
  * MNIST: $28\*28=784$ inputs for all pixels, will be either ON(1) or OFF(0). 10 outputs to represent digits 0-9
  * Initial population will be given random weights between 0 and 1
  * 10 "Fittest" networks survive each iteration, and performs crossover to reproduce 40 offspring
  * For all weights, mutation occurs to add a random value between -1 to 1, with a 5% chance
  
  With 2 hidden layers of 4 and 3 neurons respectively, XOR managed to achieve 97-99.9% accuracy in around 100 generations. Biases were not used here.
  
  **However**, trying out MNIST revealed a pretty glaring issue - the 784 inputs; a large increase of nodes compared to XOR, multiplied with weights and added up results in HUGE values of 50 to even 100, way beyond the typical domain range of the activation function (sigmoid).
  
  This just renders all layers' outputs as 1 or 0.99999-something, which breaks the entire network. Also, since this makes all individuals in a population extremely similar to one other, the genetic algorithm seems to have no clue on how to improve. The crossover will produce an offspring almost identical to its parents, and some lucky mutations are simply ignored by the sheer amount of other neurons!
  
  What can be a viable solution to this?
  
  It's my first time studying NNs, and this is really challenging.
  
  **Tags:** neural-networks, genetic-algorithms, data-preprocessing, weights-initialization, weight-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-07. Score: 4 | Accepted answer
  
  Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ones.
  
  Your weights should be both positive and negative.
  
  In addition, once trained, they tend to follow a certain size distribution. It helps if you start with values within that range. This is often called Xavier or Glorot initialisation. Basically, if your number of inputs to a layer is `n_inputs` and number of outputs is `n_outputs`, you will have `n_inputs * n_outputs` weights, and if you initialise them with a random number generator, then you should use a multiplier something like this:
  
  ```
  w = (rand() - 0.5) * sqrt(6/(n_inputs + n_outputs))
  
  ```
  
  Note that *is* an addition inside the square root, you don't take the total number of weights.
  
  As you are using a GA, you may want to use some form of clipping or normalisation to prevent mutations from drifting too far away from these useful weights. I'm not entirely sure what would be best, as usually I would use back propagation to train a large network. Potentially a maxnorm regularisation would help - choose a value (maybe even make it adjustable via mutation) that the norm of weight matrix in each layer should not exceed (separately for each layer, or one overall value, up to you), and if any learning step creates a network with too high a norm, scale it down. The L2 norm, `sqrt(sum_of_squared_weights)` is usual choice for maxnorm regularisation, and this works well with backpropagation-based supervised learning.
  
  
  ---
  
  