Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
follows: In addition to the current update, we also look at the history of updates. I encourage you to take your time to process the new update rule and try and put it on paper how the update term changes in every step. Or keep reading. Breaking it down we get You can see that the current update is