Pytorch, Machine Learning, Gradient Descent, Python, Hands On Tutorials.
functions with high momentum. Thanks to the outrageous momentum value, we can clearly see its effect on the optimizer: it overshoots the global minimum and has to swerve sloppily back around. This effect is most pronounced in the L2 loss, whose steep gradients carry it clean over the minimum and