Machine Learning, Neural Networks, Statistics, Data Science, AI.
data we have, the less regularization we might need when looking at any specific SGD step; after all, while the loss term should remain the same as m grows, so should the weights of the network, making the regularization term itself shrink in relation to the original loss term. A recent paper (and