title: ConvNets Match Vision Transformers at Scale
summary: Many researchers believe that ConvNets perform well on small or moderately
  sized datasets, but are not competitive with Vision Transformers when given
  access to datasets on the web-scale. We challenge this belief by evaluating a
  performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset
  of images often used for training foundation models. We consider pre-training
  compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a
  series of networks of increasing depth and width from the NFNet model family.
  We observe a log-log scaling law between held out loss and compute budget.
  After fine-tuning on ImageNet, NFNets match the reported performance of Vision
  Transformers with comparable compute budgets. Our strongest fine-tuned model
  achieves a Top-1 accuracy of 90.4%.
paper_authors._id: 6539dfc17174042dab924d71
paper_authors.name: Samuel L. Smith
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-04T11:27:46.510Z
paper_authors._id: 660e8dfcc6f740b010f9fb99
paper_authors.avatarUrl: /avatars/a955d7692eb7397eb58a9ffa6bc14089.svg
paper_authors.fullname: Sam Smith
paper_authors.type: user
paper_authors.user: slsmith
paper_authors._id: 6539dfc17174042dab924d72
paper_authors.name: Andrew Brock
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:24:10.454Z
paper_authors._id: 636244b92691058b19d89cca
paper_authors.avatarUrl: /avatars/97c5c27ab264c8a2d42ce50b0647b5d7.svg
paper_authors.fullname: Andrew Brock
paper_authors.type: user
paper_authors.user: Catchwa
paper_authors._id: 6539dfc17174042dab924d73
paper_authors.name: Leonard Berrada
paper_authors._id: 6539dfc17174042dab924d74
paper_authors.name: Soham De
paper_title: ConvNets Match Vision Transformers at Scale
paper_summary: Many researchers believe that ConvNets perform well on small or moderately
  sized datasets, but are not competitive with Vision Transformers when given
  access to datasets on the web-scale. We challenge this belief by evaluating a
  performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset
  of images often used for training foundation models. We consider pre-training
  compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a
  series of networks of increasing depth and width from the NFNet model family.
  We observe a log-log scaling law between held out loss and compute budget.
  After fine-tuning on ImageNet, NFNets match the reported performance of Vision
  Transformers with comparable compute budgets. Our strongest fine-tuned model
  achieves a Top-1 accuracy of 90.4%.
paper_ai_keywords: ConvNets Vision Transformers JFT-4B TPU-v4 core compute hours NFNet model family held out loss compute budget fine-tuning ImageNet Top-1 accuracy