Machine Learning, Gradient Descent, Data Visualization, Data Science.
as: delta = - learning_rate * gradient + previous_delta * decay_rate (eq. 1) theta += delta (eq. 2) I found it more intuitive if I massage this equation a little and keep track of the (decayed) cumulative sum of gradient instead. This will also make things easier when we introduce the Adam