title: When Do We Not Need Larger Vision Models?
summary: Scaling up the size of vision models has been the de facto standard to obtain
  more powerful visual representations. In this work, we discuss the point beyond
  which larger vision models are not necessary. First, we demonstrate the power
  of Scaling on Scales (S^2), whereby a pre-trained and frozen smaller vision
  model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform
  larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth
  estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation.
  Notably, S^2 achieves state-of-the-art performance in detailed understanding
  of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the
  conditions under which S^2 is a preferred scaling approach compared to
  scaling on model size. While larger models have the advantage of better
  generalization on hard examples, we show that features of larger vision models
  can be well approximated by those of multi-scale smaller models. This suggests
  most, if not all, of the representations learned by current large pre-trained
  models can also be obtained from multi-scale smaller models. Our results show
  that a multi-scale smaller model has comparable learning capacity to a larger
  model, and pre-training smaller models with S^2 can match or even exceed the
  advantage of larger models. We release a Python package that can apply S^2 on
  any vision model with one line of code:
  https://github.com/bfshi/scaling_on_scales.
paper_authors._id: 65fba11a5e18ca2ff9c7ff5a
paper_authors.name: Baifeng Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:04:40.999Z
paper_authors._id: 649004218f7cbbc94c782db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
paper_authors.fullname: Baifeng Shi
paper_authors.type: user
paper_authors.user: bfshi
paper_authors._id: 65fba11a5e18ca2ff9c7ff5b
paper_authors.name: Ziyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:04:53.849Z
paper_authors._id: 648fbaff4852c28c4cff7a3b
paper_authors.avatarUrl: /avatars/f987bb904580865676c8541839dd7615.svg
paper_authors.fullname: wuziyang
paper_authors.type: user
paper_authors.user: wzyohyes
paper_authors._id: 65fba11a5e18ca2ff9c7ff5c
paper_authors.name: Maolin Mao
paper_authors._id: 65fba11a5e18ca2ff9c7ff5d
paper_authors.name: Xin Wang
paper_authors._id: 65fba11a5e18ca2ff9c7ff5e
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:05:21.632Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_title: When Do We Not Need Larger Vision Models?
paper_summary: Scaling up the size of vision models has been the de facto standard to obtain
  more powerful visual representations. In this work, we discuss the point beyond
  which larger vision models are not necessary. First, we demonstrate the power
  of Scaling on Scales (S^2), whereby a pre-trained and frozen smaller vision
  model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform
  larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth
  estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation.
  Notably, S^2 achieves state-of-the-art performance in detailed understanding
  of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the
  conditions under which S^2 is a preferred scaling approach compared to
  scaling on model size. While larger models have the advantage of better
  generalization on hard examples, we show that features of larger vision models
  can be well approximated by those of multi-scale smaller models. This suggests
  most, if not all, of the representations learned by current large pre-trained
  models can also be obtained from multi-scale smaller models. Our results show
  that a multi-scale smaller model has comparable learning capacity to a larger
  model, and pre-training smaller models with S^2 can match or even exceed the
  advantage of larger models. We release a Python package that can apply S^2 on
  any vision model with one line of code:
  https://github.com/bfshi/scaling_on_scales.
paper_ai_summary: Running smaller vision models at multiple image scales can achieve state-of-the-art performance on various tasks, often surpassing larger models.
paper_ai_keywords: Scaling on Scales (S$^2$) pre-trained vision model ViT-B ViT-L ViT-H ViT-G classification segmentation depth estimation Multimodal LLM (MLLM) robotic manipulation V* benchmark GPT-4V learning capacity