Answers.Body: <p>In Deep Learning there are several methods to improve "stuck" gradient - decrease learning rate, use cyclic learning rate - cycle it from bigger to smaller value. More radical method is completely reinitialize last or two last (before loss) layers of the network. </p>
  
  <p>In non-Deep Learning ML out of those only decrease learning rate will work, but there is plethora Numerical Optimization methods to help, like second-order methods - variation of Gauss-Newton, or methods specific to the problem which may include projective methods, alternate directions, conjugate gradients <a href="https://en.wikipedia.org/wiki/Mathematical_optimization#Optimization_algorithms" rel="nofollow noreferrer">etc</a>. There are a lot of methods which are better then gradient descent for non-Deep Learning optimization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T07:44:20.537Z
Answers.LastActivityDate: 2019-04-09T08:35:22.947Z
Answers.LastEditDate: 2019-04-09T08:35:22.947Z
Answers.UserDisplayName: mirror2image
Body: <p>Gradient descent can get stuck into local optimum. Which techniques are there to reach global optimum?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-27T16:19:04.490Z
Comments.Text: Possible duplicate of [How to avoid falling into the "local minima" trap?](https://ai.stackexchange.com/questions/1362/how-to-avoid-falling-into-the-local-minima-trap)
Comments.UserDisplayName: nbro
Tags: <machine-learning><optimization><gradient-descent>
Title: How can we reach global optimum?
UserDisplayName: hina munir
ThreadText: # How can we reach global optimum?
  
  (Asked by: username_0 on 2019-04-09)
  
  Gradient descent can get stuck into local optimum. Which techniques are there to reach global optimum?
  
  **Tags:** machine-learning, optimization, gradient-descent
  > **Comments:**
  > **username_1**: Possible duplicate of [How to avoid falling into the "local minima" trap?](https://ai.stackexchange.com/questions/1362/how-to-avoid-falling-into-the-local-minima-trap)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-04-09. Score: 1
  
  In Deep Learning there are several methods to improve "stuck" gradient - decrease learning rate, use cyclic learning rate - cycle it from bigger to smaller value. More radical method is completely reinitialize last or two last (before loss) layers of the network.
  
  In non-Deep Learning ML out of those only decrease learning rate will work, but there is plethora Numerical Optimization methods to help, like second-order methods - variation of Gauss-Newton, or methods specific to the problem which may include projective methods, alternate directions, conjugate gradients [etc](https://en.wikipedia.org/wiki/Mathematical_optimization#Optimization_algorithms). There are a lot of methods which are better then gradient descent for non-Deep Learning optimization.
  
  
  ---
  
  