Pytorch, Machine Learning, Gradient Descent, Python, Hands On Tutorials.
Figure 8. Visualized gradient descent down all loss functions with high Nesterov momentum. When viewed graphically, we can see that Nesterov momentum has cut down the overshooting we observed with plain momentum. Especially in the L2 case, since our momentum carried us clear over the global