title: Just How Flexible are Neural Networks in Practice?
summary: It is widely believed that a neural network can fit a training set containing
  at least as many samples as it has parameters, underpinning notions of
  overparameterized and underparameterized models. In practice, however, we only
  find solutions accessible via our training procedure, including the optimizer
  and regularizers, limiting flexibility. Moreover, the exact parameterization of
  the function class, built into an architecture, shapes its loss surface and
  impacts the minima we find. In this work, we examine the ability of neural
  networks to fit data in practice. Our findings indicate that: (1) standard
  optimizers find minima where the model can only fit training sets with
  significantly fewer samples than it has parameters; (2) convolutional networks
  are more parameter-efficient than MLPs and ViTs, even on randomly labeled data;
  (3) while stochastic training is thought to have a regularizing effect, SGD
  actually finds minima that fit more training data than full-batch gradient
  descent; (4) the difference in capacity to fit correctly labeled and
  incorrectly labeled samples can be predictive of generalization; (5) ReLU
  activation functions result in finding minima that fit more data despite being
  designed to avoid vanishing and exploding gradients in deep architectures.
paper_authors._id: 66718b191ee1a34c606a95a0
paper_authors.name: Ravid Shwartz-Ziv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:28:27.898Z
paper_authors._id: 6303f7333926de1f7ec454f3
paper_authors.avatarUrl: /avatars/1677a517b8635872a7a905aac727ff59.svg
paper_authors.fullname: shwartz ziv
paper_authors.type: user
paper_authors.user: ravid
paper_authors._id: 66718b191ee1a34c606a95a1
paper_authors.name: Micah Goldblum
paper_authors._id: 66718b191ee1a34c606a95a2
paper_authors.name: Arpit Bansal
paper_authors._id: 66718b191ee1a34c606a95a3
paper_authors.name: C. Bayan Bruss
paper_authors._id: 66718b191ee1a34c606a95a4
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T16:28:45.106Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 66718b191ee1a34c606a95a5
paper_authors.name: Andrew Gordon Wilson
paper_title: Just How Flexible are Neural Networks in Practice?
paper_summary: It is widely believed that a neural network can fit a training set containing
  at least as many samples as it has parameters, underpinning notions of
  overparameterized and underparameterized models. In practice, however, we only
  find solutions accessible via our training procedure, including the optimizer
  and regularizers, limiting flexibility. Moreover, the exact parameterization of
  the function class, built into an architecture, shapes its loss surface and
  impacts the minima we find. In this work, we examine the ability of neural
  networks to fit data in practice. Our findings indicate that: (1) standard
  optimizers find minima where the model can only fit training sets with
  significantly fewer samples than it has parameters; (2) convolutional networks
  are more parameter-efficient than MLPs and ViTs, even on randomly labeled data;
  (3) while stochastic training is thought to have a regularizing effect, SGD
  actually finds minima that fit more training data than full-batch gradient
  descent; (4) the difference in capacity to fit correctly labeled and
  incorrectly labeled samples can be predictive of generalization; (5) ReLU
  activation functions result in finding minima that fit more data despite being
  designed to avoid vanishing and exploding gradients in deep architectures.
paper_ai_keywords: overparameterized models underparameterized models optimizer regularizers parameterization function class loss surface minima neural networks training sets standard optimizers convolutional networks MLPs ViTs full-batch gradient descent ReLU activations vanishing gradients exploding gradients generalization