Machine Learning, Gradient Descent, Data Visualization, Data Science.
(1- decay_rate) delta = -learning_rate * gradient / sqrt(sum_of_gradient_squared) theta += delta More precisely, the sum of gradient squared is actually the decayed sum of gradient squared. The decay rate is saying only recent gradient² matters, and the ones from long ago are basically forgotten.