Artificial Intelligence, Deep Learning, Machine Learning, Gradient Descent, Optimization.
variants because of its gradient value is boosted by the learning rate. The disadvantage of AdaGrad is that the learning rate decays very aggressively as the denominator grows which is not good for parameters corresponding to dense features. As b gets updated, again and again, denominator increases