Machine Learning, Deep Learning, Neural Networks, Optimization Algorithms, Adam.
RMSProp equations when using a squared gradient instead of the exponentially moving average As we can see, the resulting formula looks very similar to the one used in the gradient descent. However, instead of using a normal gradient value for the update, we are now using the sign of the gradient: