Machine Learning, Deep Learning, Neural Networks, Optimization Algorithms, Adam.
values. The second equation performs the normal gradient descent update using the computed moving average value on the current iteration. α is the learning rate of the algorithm. Momentum can be particularly useful for cases like the above. Imagine we have computed gradients on every iteration like