Machine Learning, Optimization, Gradient Descent, Deep Learning, Learning Parameters.
start from a random point on the error surface and look at the updates. Limitation Gradient Descent in its raw form has an obvious drawback. Let look at an example curve f(x) = x² + 1, shown below: When the curve is steep the gradient (∇y1/∇x1) is large. When the curve is gentle (∇y2/∇x2) is small.