Answers.Body: <p>Particle Swarm Optimization can be used to optimize a neural network with more than one hidden layer. Instead of optimizing a single weight matrix, and two bias vectors, you are just optimizing more of them.</p>
  <p>However, PSO is not often used for larger neural networks, because, particle swarm optimization is not all that efficient at working with a large amount of data, which you need to train larger neural networks. Particle Swarm Optimization involves taking many particles, and evaluating the fitness in all of them. If you have, let's say, 1,000,000 training examples, you are doing a lot more error calculations, even if you used a small batch size, than if you used another technique lake back-propagation or a related technique. Backpropagation and related techniques are used much more in larger neural networks because they are more efficient at working with larger sets of data.</p>
  <p>As for why the code that you have been using breaks with networks with more than one hidden layer, I cannot explain, but it is fairly easy to write your own basic implementation of PSO to train multi-layer neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-03T17:18:07.213Z
Answers.LastActivityDate: 2021-10-14T13:58:49.163Z
Answers.LastEditDate: 2021-10-14T13:58:49.163Z
Answers.UserDisplayName: Aiden Grossman
Body: <p>I've been thinking about the idea of replacing the classic gradient descent algorithm with an algorithm that is less sensitive to a local optimum. I was thinking about particle swarm optimization (PSO), which thus tries to select the best weights and biases for the model.</p>
  <p>But I've seen everywhere that only one hidden layer is used (no one explains why just one layer is being used) and all those codes break when I try to use more than one hidden layer, so the questions are:</p>
  <ol>
  <li><p>Can't PSO be used to optimize an Artificial Neural Network with more than one hidden layer?</p>
  </li>
  <li><p>In that case, why is that?</p>
  </li>
  </ol>
  
Tags: <neural-networks><deep-learning><training><hidden-layers><particle-swarm-optimization>
Title: Can particle swarm optimization be used to train neural networks with more than one hidden layer?
UserDisplayName: gerard
ThreadText: # Can particle swarm optimization be used to train neural networks with more than one hidden layer?
  
  (Asked by: username_0 on 2019-03-03)
  
  I've been thinking about the idea of replacing the classic gradient descent algorithm with an algorithm that is less sensitive to a local optimum. I was thinking about particle swarm optimization (PSO), which thus tries to select the best weights and biases for the model.
  
  But I've seen everywhere that only one hidden layer is used (no one explains why just one layer is being used) and all those codes break when I try to use more than one hidden layer, so the questions are:
  
  1. Can't PSO be used to optimize an Artificial Neural Network with more than one hidden layer?
  2. In that case, why is that?
  
  **Tags:** neural-networks, deep-learning, training, hidden-layers, particle-swarm-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-03. Score: 1 | Accepted answer
  
  Particle Swarm Optimization can be used to optimize a neural network with more than one hidden layer. Instead of optimizing a single weight matrix, and two bias vectors, you are just optimizing more of them.
  
  However, PSO is not often used for larger neural networks, because, particle swarm optimization is not all that efficient at working with a large amount of data, which you need to train larger neural networks. Particle Swarm Optimization involves taking many particles, and evaluating the fitness in all of them. If you have, let's say, 1,000,000 training examples, you are doing a lot more error calculations, even if you used a small batch size, than if you used another technique lake back-propagation or a related technique. Backpropagation and related techniques are used much more in larger neural networks because they are more efficient at working with larger sets of data.
  
  As for why the code that you have been using breaks with networks with more than one hidden layer, I cannot explain, but it is fairly easy to write your own basic implementation of PSO to train multi-layer neural networks.
  
  
  ---
  
  