Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
high on either side of the minima valley. Could momentum still work well in such cases or could it be detrimental instead? Let’s find out. 100 iterations of vanilla gradient descent make the black patch. We can observe that momentum-based gradient descent oscillates in and out of the minima valley