ConvNets Match Vision Transformers at Scale
Many researchers believe that ConvNets perform well on small or moderately
sized datasets, but are not competitive with Vision Transformers when given
access to datasets on the web-scale. We challenge this belief by evaluating a
performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset
of images often used for training foundation models. We consider pre-training
compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a
series of networks of increasing depth and width from the NFNet model family.
We observe a log-log scaling law between held out loss and compute budget.
After fine-tuning on ImageNet, NFNets match the reported performance of Vision
Transformers with comparable compute budgets. Our strongest fine-tuned model
achieves a Top-1 accuracy of 90.4%.
6539dfc17174042dab924d71
Samuel L. Smith
extracted_confirmed
2024-04-04T11:27:46.510Z
660e8dfcc6f740b010f9fb99
/avatars/a955d7692eb7397eb58a9ffa6bc14089.svg
Sam Smith
user
slsmith
6539dfc17174042dab924d72
Andrew Brock
admin_assigned
2023-10-26T08:24:10.454Z
636244b92691058b19d89cca
/avatars/97c5c27ab264c8a2d42ce50b0647b5d7.svg
Andrew Brock
user
Catchwa
6539dfc17174042dab924d73
Leonard Berrada
6539dfc17174042dab924d74
Soham De
claimed_verified
2025-04-28T13:50:48.056Z
661415d9f9e58fe02bd8661d
/avatars/7a28bb507b8bac72a8b161e774b19fbd.svg
Soham De
user
sohamde
ConvNets Match Vision Transformers at Scale
Many researchers believe that ConvNets perform well on small or moderately
sized datasets, but are not competitive with Vision Transformers when given
access to datasets on the web-scale. We challenge this belief by evaluating a
performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset
of images often used for training foundation models. We consider pre-training
compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a
series of networks of increasing depth and width from the NFNet model family.
We observe a log-log scaling law between held out loss and compute budget.
After fine-tuning on ImageNet, NFNets match the reported performance of Vision
Transformers with comparable compute budgets. Our strongest fine-tuned model
achieves a Top-1 accuracy of 90.4%.
ConvNets pre-trained on a large dataset match the performance of Vision Transformers on ImageNet with comparable computational resources.
ConvNets Vision Transformers JFT-4B pre-training TPU-v4 NFNet log-log scaling law fine-tuning ImageNet Top-1 accuracy