Machine Learning, Gradient Descent, Data Visualization, Data Science.
point that AdaGrad’s path is straighter stays largely true regardless of learning rate. This property allows AdaGrad (and other similar gradient-squared-based methods like RMSProp and Adam) to escape a saddle point much better. AdaGrad will take a straight path, whereas gradient descent (or