Machine Learning, Neural Networks, Statistics, Data Science, AI.
side-effect of batch gradient descent Let’s look at this from the point of view of weight decay. In the stochastic variant of gradient descent (SGD), we evaluate the gradient of the loss function (in respect to parameters θ) over a single training example at a time. If we think of weight decay as