Are deep neural networks dramatically overfitted? (2019)
I get so pissed every time the double descent paper gets brought up, maybe because of the hubris in its abstract. No "questions about the mathematical foundation of machine learning" were raised by deep learning and a few silly experiments are far from showing "limits of classical analyses".<p>All the paper shows is (1) stupid ways of counting model complexity and (2) that gradient descent is flawed. Nobody in their right mind believes that <i>increasing</i> the number of hidden neurons can result in a network that is <i>worse</i> on the test set. Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot find it. But of course "SGD cannot always find good solutions" is surprising to nobody, so let's just shit on decades of serious work to get our little paper out.<p>Sorry for the rant.