Machine Learning, Neural Networks, Statistics, Data Science, AI.
with the gradient ∇J in respect to w, we also subtract from it λ∙w. This gives the weights a tendency to decay towards zero, hence the name. This is actually quite an early concept in the history of deep learning. Simplifying neural networks by soft weight sharing, a paper from 1992 by Nowlan and