Artificial Intelligence, Deep Learning, Machine Learning, Gradient Descent, Optimization.
compare AdaGrad update rule with Vanilla GD (black curve), Momentum GD (red), NAG (blue) and AdaGrad (green) Comparison of different GD variants From the above plot, we can see that AdaGrad is taking larger steps in the direction of w even though it is a sparse feature in contrast to the other