Optimization, Learning Parameters, Gradient Descent, Stochastic Gradient, Deep Learning.
is locally favorable to one point may harm other points (its almost as if the data points are competing with each other). Can we reduce the oscillations by improving our stochastic estimates of the gradient (currently estimated from just 1 data point at a time)? Yes, let’s look at mini-batch