title: Battle of the Backbones: A Large-Scale Comparison of Pretrained Models
    across Computer Vision Tasks
summary: Neural network based computer vision systems are typically built on a
  backbone, a pretrained or randomly initialized feature extractor. Several years
  ago, the default option was an ImageNet-trained convolutional neural network.
  However, the recent past has seen the emergence of countless backbones
  pretrained using various algorithms and datasets. While this abundance of
  choice has led to performance increases for a range of systems, it is difficult
  for practitioners to make informed decisions about which backbone to choose.
  Battle of the Backbones (BoB) makes this choice easier by benchmarking a
  diverse suite of pretrained models, including vision-language models, those
  trained via self-supervised learning, and the Stable Diffusion backbone, across
  a diverse set of computer vision tasks ranging from classification to object
  detection to OOD generalization and more. Furthermore, BoB sheds light on
  promising directions for the research community to advance computer vision by
  illuminating strengths and weakness of existing approaches through a
  comprehensive analysis conducted on more than 1500 training runs. While vision
  transformers (ViTs) and self-supervised learning (SSL) are increasingly
  popular, we find that convolutional neural networks pretrained in a supervised
  fashion on large training sets still perform best on most tasks among the
  models we consider. Moreover, in apples-to-apples comparisons on the same
  architectures and similarly sized pretraining datasets, we find that SSL
  backbones are highly competitive, indicating that future works should perform
  SSL pretraining with advanced architectures and larger pretraining datasets. We
  release the raw results of our experiments along with code that allows
  researchers to put their own backbones through the gauntlet here:
  https://github.com/hsouri/Battle-of-the-Backbones
paper_authors._id: 6541b949b887e805a441ebc2
paper_authors.name: Micah Goldblum
paper_authors._id: 6541b949b887e805a441ebc3
paper_authors.name: Hossein Souri
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-28T16:16:09.450Z
paper_authors._id: 6317bf031d6018cb85185870
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6317bf031d6018cb85185870/cZpMcYaMVZkUEuuG1DXkI.png
paper_authors.fullname: Hossein Souri
paper_authors.type: user
paper_authors.user: hsouri
paper_authors._id: 6541b949b887e805a441ebc4
paper_authors.name: Renkun Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:32:13.771Z
paper_authors._id: 637a79ea3f8f451d6bd50580
paper_authors.avatarUrl: /avatars/80618daa0fcb3ac9e4387f385e40824f.svg
paper_authors.fullname: Renkun Ni
paper_authors.type: user
paper_authors.user: rn9zm
paper_authors._id: 6541b949b887e805a441ebc5
paper_authors.name: Manli Shu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:32:21.639Z
paper_authors._id: 63bdcb129a15a3e94198e7d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
paper_authors.fullname: Manli Shu
paper_authors.type: user
paper_authors.user: Manli
paper_authors._id: 6541b949b887e805a441ebc6
paper_authors.name: Viraj Prabhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:32:32.150Z
paper_authors._id: 63fe4f92c0ec83fda436b6bf
paper_authors.avatarUrl: /avatars/1021060e042ce3257444577724aa9d34.svg
paper_authors.fullname: Viraj Prabhu
paper_authors.type: user
paper_authors.user: virajp
paper_authors._id: 6541b949b887e805a441ebc7
paper_authors.name: Gowthami Somepalli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:32:39.789Z
paper_authors._id: 62ae853a4b5d6165083a12f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ae853a4b5d6165083a12f8/3_Q1yB3Qfpd9PngzzBlgZ.jpeg
paper_authors.fullname: Gowthami Somepalli
paper_authors.type: user
paper_authors.user: somepago
paper_authors._id: 6541b949b887e805a441ebc8
paper_authors.name: Prithvijit Chattopadhyay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-02T11:34:59.837Z
paper_authors._id: 628d451386d23ad1560882c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628d451386d23ad1560882c4/UMxez0DEvX5qdP5ddqi-8.png
paper_authors.fullname: Prithvijit Chattopadhyay
paper_authors.type: user
paper_authors.user: prithv1
paper_authors._id: 6541b949b887e805a441ebc9
paper_authors.name: Mark Ibrahim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T15:20:25.645Z
paper_authors._id: 64c17345e82e55936cf971bc
paper_authors.avatarUrl: /avatars/b39c5637e00291410620afb7c770d587.svg
paper_authors.fullname: Mark Ibrahim
paper_authors.type: user
paper_authors.user: marksibrahim
paper_authors._id: 6541b949b887e805a441ebca
paper_authors.name: Adrien Bardes
paper_authors._id: 6541b949b887e805a441ebcb
paper_authors.name: Judy Hoffman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:32:54.177Z
paper_authors._id: 6477790c4ae93470ffc297bb
paper_authors.avatarUrl: /avatars/f8d5e08719297a335731a117416d6336.svg
paper_authors.fullname: Judy Hoffman
paper_authors.type: user
paper_authors.user: judyhoffman
paper_authors._id: 6541b949b887e805a441ebcc
paper_authors.name: Rama Chellappa
paper_authors._id: 6541b949b887e805a441ebcd
paper_authors.name: Andrew Gordon Wilson
paper_authors._id: 6541b949b887e805a441ebce
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:33:09.360Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_title: Battle of the Backbones: A Large-Scale Comparison of Pretrained Models
    across Computer Vision Tasks
paper_summary: Neural network based computer vision systems are typically built on a
  backbone, a pretrained or randomly initialized feature extractor. Several years
  ago, the default option was an ImageNet-trained convolutional neural network.
  However, the recent past has seen the emergence of countless backbones
  pretrained using various algorithms and datasets. While this abundance of
  choice has led to performance increases for a range of systems, it is difficult
  for practitioners to make informed decisions about which backbone to choose.
  Battle of the Backbones (BoB) makes this choice easier by benchmarking a
  diverse suite of pretrained models, including vision-language models, those
  trained via self-supervised learning, and the Stable Diffusion backbone, across
  a diverse set of computer vision tasks ranging from classification to object
  detection to OOD generalization and more. Furthermore, BoB sheds light on
  promising directions for the research community to advance computer vision by
  illuminating strengths and weakness of existing approaches through a
  comprehensive analysis conducted on more than 1500 training runs. While vision
  transformers (ViTs) and self-supervised learning (SSL) are increasingly
  popular, we find that convolutional neural networks pretrained in a supervised
  fashion on large training sets still perform best on most tasks among the
  models we consider. Moreover, in apples-to-apples comparisons on the same
  architectures and similarly sized pretraining datasets, we find that SSL
  backbones are highly competitive, indicating that future works should perform
  SSL pretraining with advanced architectures and larger pretraining datasets. We
  release the raw results of our experiments along with code that allows
  researchers to put their own backbones through the gauntlet here:
  https://github.com/hsouri/Battle-of-the-Backbones
paper_ai_keywords: neural network backbone pretrained ImageNet-trained convolutional neural network self-supervised learning Stable Diffusion vision-language models object detection OOD generalization battle of the backbones vision transformers (ViTs) SSL supervised fashion architecture