Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
overshoot but a smaller one when compared to the vanilla momentum-based gradient descent. And that my friend, is how NAG helps us in reducing the overshoots, i.e. making us take shorter U-turns. NAG In Action By updating the momentum code slightly to do both the partial update and the full update,