Deep Learning, Optimization, Gradient Descent, Momentum, Learning Parameters.
But why would looking ahead help us in avoiding overshoots? I urge you to pause and ponder. If it’s not clear, I am sure it will be clear in the next few minutes. Take a look at this figure for a moment. A toy illustration. In figure (a), update 1 is positive i.e., the gradient is negative because