Proving the Lottery Ticket Hypothesis: Pruning is All You Need
The lottery ticket hypothesis is IMHO the single most interesting finding for deep learning. It explains why does deep learning works (vs shallow neural nets), why is initial over-parametrization is often useful, why deeper is often better than shallow, etc.<p>I recommend for an overview:<p>- the original paper "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", <a href="https://arxiv.org/abs/1803.03635" rel="nofollow">https://arxiv.org/abs/1803.03635</a><p>- "Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask" <a href="https://eng.uber.com/deconstructing-lottery-tickets/" rel="nofollow">https://eng.uber.com/deconstructing-lottery-tickets/</a> showing that if we remove "non-winning tickets" before the training, the trained network still works well