Machine Learning, Gradient Descent, Data Visualization, Data Science.
previous_sum_of_gradient * beta1 + gradient * (1 - beta1) [Momentum] sum_of_gradient_squared = previous_sum_of_gradient_squared * beta2 + gradient² * (1- beta2) [RMSProp] delta = -learning_rate * sum_of_gradient / sqrt(sum_of_gradient_squared) theta += delta Beta1 is the decay rate for the first