Optimization, Learning Parameters, Gradient Descent, Stochastic Gradient, Deep Learning.
we will make a million updates in each epoch (1 epoch = 1 pass over the data; 1 step = 1 update). What is the flipside? It is an approximate (rather stochastic) gradient so no guarantee that each step will decrease the loss. SGD In Action Let us see this algorithm geometrically when we have a few