Optimization, Learning Parameters, Gradient Descent, Stochastic Gradient, Deep Learning.
NAG We can have stochastic versions of momentum based gradient descent and Nesterov accelerated gradient descent. While the stochastic versions of both Momentum (red)and NAG (blue) exhibit oscillations the relative advantage of NAG over Momentum still holds (i.e., NAG takes relatively shorter