Optimization, Learning Parameters, Gradient Descent, Stochastic Gradient, Deep Learning.
Basically, you are going with an approximation of some sort instead of the noble ‘true gradient’. Stochastic gradient descent is the dominant method used to train deep learning models. Let’s straightaway look at the code for SGD. Here’s what’s going on — instead of making an update by calculating