Answers.Body: <p>There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include:</p>
  
  <ul>
  <li>Probabalistically accepting worse solutions in the hope that this
  will jump out of the current basin (like Metropolis-Hastings acceptance in Simulated Annealing). </li>
  <li>Maintaining a list of recently-encountered states (or attributes thereof) and not returning
  to a recently-encountered one (like Tabu Search). </li>
  <li>Performing a random walk of a length determined by the current state of the search (an explicit 'Diversification strategy', e.g. as used in 'Reactive Tabu Search').</li>
  </ul>
  
  <p>See the excellent (and free online) book <a href="https://cs.gmu.edu/~sean/book/metaheuristics/">'Essentials of Metaheuristics'</a> by Sean Luke for more details on these kind of techniques and some rules of thumb about when and how to use them.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-05T11:00:14.427Z
Answers.LastActivityDate: 2016-08-05T11:00:14.427Z
Answers.UserDisplayName: NietzscheanAI
Body: <p>How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network?</p>
  
  <p>Are there any methods which help me avoid it?</p>
  
Tags: <neural-networks><backpropagation><optimization><gradient-descent>
Title: How to avoid falling into the "local minima" trap?
UserDisplayName: Dawny33
ThreadText: # How to avoid falling into the "local minima" trap?
  
  (Asked by: username_0 on 2016-08-05)
  
  How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network?
  
  Are there any methods which help me avoid it?
  
  **Tags:** neural-networks, backpropagation, optimization, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-08-05. Score: 9 | Accepted answer
  
  There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include:
  
  * Probabalistically accepting worse solutions in the hope that this
    will jump out of the current basin (like Metropolis-Hastings acceptance in Simulated Annealing).
  * Maintaining a list of recently-encountered states (or attributes thereof) and not returning
    to a recently-encountered one (like Tabu Search).
  * Performing a random walk of a length determined by the current state of the search (an explicit 'Diversification strategy', e.g. as used in 'Reactive Tabu Search').
  
  See the excellent (and free online) book ['Essentials of Metaheuristics'](https://cs.gmu.edu/~sean/book/metaheuristics/) by Sean Luke for more details on these kind of techniques and some rules of thumb about when and how to use them.
  
  
  ---
  
  