Machine Learning, Optimization, Gradient Descent, Deep Learning, Learning Parameters.
Recall that our weight updates are proportional to the gradient w = w — η∇w. Hence, in the regions where the curve is gentle, the updates are small whereas in the regions where the curve is steep the updates are large. Let’s see what happens when we start from a different point on the surface.