Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
proportional to not just the present gradient but also gradients of previous steps, although their contribution reduces every time step by γ(gamma) times. And that is how we boost the magnitude of the update at gentle regions. Momentum In Action We modify the vanilla gradient descent code (shown in