Machine Learning, Optimization, Gradient Descent, Deep Learning, Learning Parameters.
Gradient Descent overcomes this drawback to an extent by letting the ‘momentum’ of recent gradient updates control the update magnitude in the current step. Read all about it in the next post of this series at: Learning Parameters, Part 2: Momentum-Based And Nesterov Accelerated Gradient Descent