Answers.Body: <p>There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing, and several others. Almost any optimization algorithm can be used to train a neural network. Here is an overview of some of the algorithms I listed:</p>
  
  <ul>
  <li>Particle Swarm optimization - I would say that this is one of the better optimization algorithms to train neural networks other than back propagation. I am currently using it and have achieved quite good results.</li>
  <li>Genetic Algorithms - I have tried to use genetic algorithms to train neural networks in the past and I was not able to get it to work. However, I was using deep neural networks with almost a million parameters and the performance was not that good.</li>
  <li>Simulated annealing - simulated annealing is based off of metals cooling. I have seen simulated annealing work fairly well but maybe not as well as particle swarm optimization.</li>
  <li>Derivatives of genetic algorithms - derivatives of genetic algorithms such as NEAT have been shown to work pretty well. I have not personally used them extensively but some of the things that people have used them for are pretty cool.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-25T20:12:12.767Z
Answers.LastActivityDate: 2017-08-25T20:17:42.777Z
Answers.LastEditDate: 2017-08-25T20:17:42.777Z
Answers.UserDisplayName: Aiden Grossman
Body: <p>As I know, the current state of the art methods for training deep learning networks are variants of gradient descent or stochastic gradient descent.  </p>
  
  <p>What are the best known gradient-free training methods for deep learning (mostly in visual tasks context)?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-30T13:53:03.703Z
Comments.Text: Several algorithms are mentioned in this slide of class of M.Kochenderfer http://adl.stanford.edu/aa222/Lecture_Notes_files/chapter6_gradfree.pdf (p.s. The problem is more general then Deep Learning)
Comments.UserDisplayName: Konstantin Burlachenko
Tags: <deep-learning><reference-request><training><algorithm-request>
Title: What are the best known gradient-free training methods for deep learning?
UserDisplayName: rkellerm
ThreadText: # What are the best known gradient-free training methods for deep learning?
  
  (Asked by: username_0 on 2017-08-24)
  
  As I know, the current state of the art methods for training deep learning networks are variants of gradient descent or stochastic gradient descent.
  
  What are the best known gradient-free training methods for deep learning (mostly in visual tasks context)?
  
  **Tags:** deep-learning, reference-request, training, algorithm-request
  > **Comments:**
  > **username_1**: Several algorithms are mentioned in this slide of class of M.Kochenderfer http://adl.stanford.edu/aa222/Lecture_Notes_files/chapter6_gradfree.pdf (p.s. The problem is more general then Deep Learning)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-08-25. Score: 8 | Accepted answer
  
  There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing, and several others. Almost any optimization algorithm can be used to train a neural network. Here is an overview of some of the algorithms I listed:
  
  * Particle Swarm optimization - I would say that this is one of the better optimization algorithms to train neural networks other than back propagation. I am currently using it and have achieved quite good results.
  * Genetic Algorithms - I have tried to use genetic algorithms to train neural networks in the past and I was not able to get it to work. However, I was using deep neural networks with almost a million parameters and the performance was not that good.
  * Simulated annealing - simulated annealing is based off of metals cooling. I have seen simulated annealing work fairly well but maybe not as well as particle swarm optimization.
  * Derivatives of genetic algorithms - derivatives of genetic algorithms such as NEAT have been shown to work pretty well. I have not personally used them extensively but some of the things that people have used them for are pretty cool.
  
  
  ---
  
  