Machine Learning, Deep Learning, Neural Networks, Optimization Algorithms, Adam.
Optimization with AdaGrad RMSProp (Root Mean Square Propagation) RMSProp was elaborated as an improvement over AdaGrad which tackles the issue of learning rate decay. Similarly to AdaGrad, RMSProp uses a pair of equations for which the weight update is absolutely the same. RMSProp equations