Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
Update 4 is where things get interesting. In vanilla momentum case, due to the positive history, the update overshoots and the descent recovers by doing negative updates. But in NAG’s case, every update happens in two steps — first, a partial update, where we get to the look_ahead point and then