Pytorch, Machine Learning, Gradient Descent, Python, Hands On Tutorials.
bring it very close to diverging. Nesterov Momentum Nesterov momentum is an interesting tweak on momentum. Normal momentum adds in some of the gradient from the last step to the gradient for the current step, giving us the scenario in figure 7(a) below. But if we already know where the gradient