Machine Learning, Deep Learning, Neural Networks, Optimization, Towards Data Science.
defines it in his Deep Learning Specialization on coursera. The way he explains it, we define a momentum, which is a moving average of our gradients. We then use it to update the weight of the network. This could written as follows: Where L — is loss function, triangular thing — gradient w.r.t