Machine Learning, Gradient Descent, Data Visualization, Data Science.
addition to decaying, the sum of gradient squared will be sqrt(1 - 0.99) = 0.1 that of AdaGrad, and thus the step is on the order of 10x larger for the same learning rate. RMSProp (green) vs AdaGrad (white). The first run just shows the balls; the second run also shows the sum of gradient squared