title: Scaling MLPs: A Tale of Inductive Bias
summary: In this work we revisit the most fundamental building block in deep learning,
  the multi-layer perceptron (MLP), and study the limits of its performance on
  vision tasks. Empirical insights into MLPs are important for multiple reasons.
  (1) Given the recent narrative "less inductive bias is better", popularized due
  to transformers eclipsing convolutional models, it is natural to explore the
  limits of this hypothesis. To that end, MLPs offer an ideal test bed, being
  completely free of any inductive bias. (2) MLPs have almost exclusively been
  the main protagonist in the deep learning theory literature due to their
  mathematical simplicity, serving as a proxy to explain empirical phenomena
  observed for more complex architectures. Surprisingly, experimental datapoints
  for MLPs are very difficult to find in the literature, especially when coupled
  with large pre-training protocols. This discrepancy between practice and theory
  is worrying: Do MLPs reflect the empirical advances exhibited by practical
  models? Or do theorists need to rethink the role of MLPs as a proxy? We provide
  insights into both these aspects. We show that the performance of MLPs
  drastically improves with scale (93% on CIFAR10, 79% on CIFAR100, 69% on
  TinyImageNet), highlighting that lack of inductive bias can indeed be
  compensated. We observe that MLPs mimic the behaviour of their modern
  counterparts faithfully, with some components in the learning setting however
  surprisingly exhibiting stronger or unexpected behaviours. Due to their
  inherent computational efficiency, large pre-training experiments become more
  accessible for academic researchers. All of our experiments were run on a
  single GPU.
paper_authors._id: 64991d0276d49ee00f7c43f5
paper_authors.name: Gregor Bachmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-27T07:27:57.277Z
paper_authors._id: 64996389a3f227b05cbd956f
paper_authors.avatarUrl: /avatars/a586e7f99efdc7b61e05d62945575096.svg
paper_authors.fullname: Gregor Bachmann
paper_authors.type: user
paper_authors.user: gregorbachmann
paper_authors._id: 64991d0276d49ee00f7c43f6
paper_authors.name: Sotiris Anagnostidis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T07:37:24.102Z
paper_authors._id: 62f8f4ff92e64c61bc6938da
paper_authors.avatarUrl: /avatars/d386eb35d2c3d52186b2a8ec957f51bc.svg
paper_authors.fullname: Sotiris Anagnostidis
paper_authors.type: user
paper_authors.user: sanagnos
paper_authors._id: 64991d0276d49ee00f7c43f7
paper_authors.name: Thomas Hofmann
paper_title: Scaling MLPs: A Tale of Inductive Bias
paper_summary: In this work we revisit the most fundamental building block in deep learning,
  the multi-layer perceptron (MLP), and study the limits of its performance on
  vision tasks. Empirical insights into MLPs are important for multiple reasons.
  (1) Given the recent narrative "less inductive bias is better", popularized due
  to transformers eclipsing convolutional models, it is natural to explore the
  limits of this hypothesis. To that end, MLPs offer an ideal test bed, being
  completely free of any inductive bias. (2) MLPs have almost exclusively been
  the main protagonist in the deep learning theory literature due to their
  mathematical simplicity, serving as a proxy to explain empirical phenomena
  observed for more complex architectures. Surprisingly, experimental datapoints
  for MLPs are very difficult to find in the literature, especially when coupled
  with large pre-training protocols. This discrepancy between practice and theory
  is worrying: Do MLPs reflect the empirical advances exhibited by practical
  models? Or do theorists need to rethink the role of MLPs as a proxy? We provide
  insights into both these aspects. We show that the performance of MLPs
  drastically improves with scale (93% on CIFAR10, 79% on CIFAR100, 69% on
  TinyImageNet), highlighting that lack of inductive bias can indeed be
  compensated. We observe that MLPs mimic the behaviour of their modern
  counterparts faithfully, with some components in the learning setting however
  surprisingly exhibiting stronger or unexpected behaviours. Due to their
  inherent computational efficiency, large pre-training experiments become more
  accessible for academic researchers. All of our experiments were run on a
  single GPU.
paper_ai_keywords: multi-layer perceptron (MLP) inductive bias transformers convolutional models vision tasks deep learning theory literature CIFAR10 CIFAR100 TinyImageNet large pre-training protocols computational efficiency