When Do We Not Need Larger Vision Models?
Scaling up the size of vision models has been the de facto standard to obtain
more powerful visual representations. In this work, we discuss the point beyond
which larger vision models are not necessary. First, we demonstrate the power
of Scaling on Scales (S^2), whereby a pre-trained and frozen smaller vision
model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform
larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth
estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation.
Notably, S^2 achieves state-of-the-art performance in detailed understanding
of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the
conditions under which S^2 is a preferred scaling approach compared to
scaling on model size. While larger models have the advantage of better
generalization on hard examples, we show that features of larger vision models
can be well approximated by those of multi-scale smaller models. This suggests
most, if not all, of the representations learned by current large pre-trained
models can also be obtained from multi-scale smaller models. Our results show
that a multi-scale smaller model has comparable learning capacity to a larger
model, and pre-training smaller models with S^2 can match or even exceed the
advantage of larger models. We release a Python package that can apply S^2 on
any vision model with one line of code:
https://github.com/bfshi/scaling_on_scales.
65fba11a5e18ca2ff9c7ff5a
Baifeng Shi
admin_assigned
2024-03-21T09:04:40.999Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
65fba11a5e18ca2ff9c7ff5b
Ziyang Wu
admin_assigned
2024-03-21T09:04:53.849Z
648fbaff4852c28c4cff7a3b
/avatars/f987bb904580865676c8541839dd7615.svg
wuziyang
user
wzyohyes
65fba11a5e18ca2ff9c7ff5c
Maolin Mao
65fba11a5e18ca2ff9c7ff5d
Xin Wang
65fba11a5e18ca2ff9c7ff5e
Trevor Darrell
admin_assigned
2024-03-21T09:05:21.632Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
When Do We Not Need Larger Vision Models?
Scaling up the size of vision models has been the de facto standard to obtain
more powerful visual representations. In this work, we discuss the point beyond
which larger vision models are not necessary. First, we demonstrate the power
of Scaling on Scales (S^2), whereby a pre-trained and frozen smaller vision
model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform
larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth
estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation.
Notably, S^2 achieves state-of-the-art performance in detailed understanding
of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the
conditions under which S^2 is a preferred scaling approach compared to
scaling on model size. While larger models have the advantage of better
generalization on hard examples, we show that features of larger vision models
can be well approximated by those of multi-scale smaller models. This suggests
most, if not all, of the representations learned by current large pre-trained
models can also be obtained from multi-scale smaller models. Our results show
that a multi-scale smaller model has comparable learning capacity to a larger
model, and pre-training smaller models with S^2 can match or even exceed the
advantage of larger models. We release a Python package that can apply S^2 on
any vision model with one line of code:
https://github.com/bfshi/scaling_on_scales.
Running smaller vision models at multiple image scales can achieve state-of-the-art performance on various tasks, often surpassing larger models.
Scaling on Scales (S$^2$) pre-trained vision model ViT-B ViT-L ViT-H ViT-G classification segmentation depth estimation Multimodal LLM (MLLM) robotic manipulation V* benchmark GPT-4V learning capacity