Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
we get the code for NAG. Let’s compare the convergence of momentum-based method with NAG using the same toy example and the same error surface we used while illustrating the momentum-based method. You can see that (I hope you can) NAG (blue) is taking smaller U-turns compared to vanilla momentum