Optimization, Learning Parameters, Gradient Descent, Stochastic Gradient, Deep Learning.
there as long as we are using an approximate gradient as opposed to the ‘true gradient.’ The illustration isn’t clear, and on top of that the k is just 2, so it is hard to see much of a difference. But trust the math, mini-batch helps us get slightly better gradient estimates. SGD With Momentum And