<p>In Deep Learning there are several methods to improve "stuck" gradient - decrease learning rate, use cyclic learning rate - cycle it from bigger to smaller value. More radical method is completely reinitialize last or two last (before loss) layers of the network. </p>

<p>In non-Deep Learning ML out of those only decrease learning rate will work, but there is plethora Numerical Optimization methods to help, like second-order methods - variation of Gauss-Newton, or methods specific to the problem which may include projective methods, alternate directions, conjugate gradients <a href="https://en.wikipedia.org/wiki/Mathematical_optimization#Optimization_algorithms" rel="nofollow noreferrer">etc</a>. There are a lot of methods which are better then gradient descent for non-Deep Learning optimization.</p>

CC BY-SA 4.0
2019-04-09T07:44:20.537Z
2019-04-09T08:35:22.947Z
2019-04-09T08:35:22.947Z
mirror2image
<p>Gradient descent can get stuck into local optimum. Which techniques are there to reach global optimum?</p>

CC BY-SA 4.0
2019-04-27T16:19:04.490Z
Possible duplicate of [How to avoid falling into the "local minima" trap?](https://ai.stackexchange.com/questions/1362/how-to-avoid-falling-into-the-local-minima-trap)
nbro
<machine-learning><optimization><gradient-descent>
How can we reach global optimum?
hina munir
# How can we reach global optimum?

(Asked by: username_0 on 2019-04-09)

Gradient descent can get stuck into local optimum. Which techniques are there to reach global optimum?

**Tags:** machine-learning, optimization, gradient-descent

> **Comments:**
> **username_1**: Possible duplicate of [How to avoid falling into the "local minima" trap?](https://ai.stackexchange.com/questions/1362/how-to-avoid-falling-into-the-local-minima-trap)  

---

## 1 answers

### Answer by username_2 on 2019-04-09. Score: 1

In Deep Learning there are several methods to improve "stuck" gradient - decrease learning rate, use cyclic learning rate - cycle it from bigger to smaller value. More radical method is completely reinitialize last or two last (before loss) layers of the network.

In non-Deep Learning ML out of those only decrease learning rate will work, but there is plethora Numerical Optimization methods to help, like second-order methods - variation of Gauss-Newton, or methods specific to the problem which may include projective methods, alternate directions, conjugate gradients [etc](https://en.wikipedia.org/wiki/Mathematical_optimization#Optimization_algorithms). There are a lot of methods which are better then gradient descent for non-Deep Learning optimization.


---

