Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Since the introduction of Vision Transformer (ViT), patchification has long
been regarded as a de facto image tokenization approach for plain visual
architectures. By compressing the spatial size of images, this approach can
effectively shorten the token sequence and reduce the computational cost of
ViT-like plain architectures. In this work, we aim to thoroughly examine the
information loss caused by this patchification-based compressive encoding
paradigm and how it affects visual understanding. We conduct extensive patch
size scaling experiments and excitedly observe an intriguing scaling law in
patchification: the models can consistently benefit from decreased patch sizes
and attain improved predictive performance, until it reaches the minimum patch
size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable
across different vision tasks, various input scales, and diverse architectures
such as ViT and the recent Mamba models. Moreover, as a by-product, we discover
that with smaller patches, task-specific decoder heads become less critical for
dense prediction. In the experiments, we successfully scale up the visual
sequence to an exceptional length of 50,176 tokens, achieving a competitive
test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We
hope this study can provide insights and theoretical foundations for future
works of building non-compressive vision models. Code is available at
https://github.com/wangf3014/Patch_Scaling.
67a8d049406cb5a65f847eb1
Feng Wang
67a8d049406cb5a65f847eb2
Yaodong Yu
admin_assigned
2025-02-10T16:22:16.391Z
6100e69a393be1b5c4c83867
/avatars/1b87098cffb9c50345789808daea4f68.svg
Yaodong Yu
user
yaodongyu
67a8d049406cb5a65f847eb3
Guoyizhe Wei
67a8d049406cb5a65f847eb4
Wei Shao
67a8d049406cb5a65f847eb5
Yuyin Zhou
admin_assigned
2025-02-10T16:22:41.354Z
66c7fb4ce2c92fe5b132f314
/avatars/22d915fa339a70803c5c748255250256.svg
Yuyin Zhou
user
RitaCoding
67a8d049406cb5a65f847eb6
Alan Yuille
67a8d049406cb5a65f847eb7
Cihang Xie
admin_assigned
2025-02-10T16:22:31.527Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Since the introduction of Vision Transformer (ViT), patchification has long
been regarded as a de facto image tokenization approach for plain visual
architectures. By compressing the spatial size of images, this approach can
effectively shorten the token sequence and reduce the computational cost of
ViT-like plain architectures. In this work, we aim to thoroughly examine the
information loss caused by this patchification-based compressive encoding
paradigm and how it affects visual understanding. We conduct extensive patch
size scaling experiments and excitedly observe an intriguing scaling law in
patchification: the models can consistently benefit from decreased patch sizes
and attain improved predictive performance, until it reaches the minimum patch
size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable
across different vision tasks, various input scales, and diverse architectures
such as ViT and the recent Mamba models. Moreover, as a by-product, we discover
that with smaller patches, task-specific decoder heads become less critical for
dense prediction. In the experiments, we successfully scale up the visual
sequence to an exceptional length of 50,176 tokens, achieving a competitive
test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We
hope this study can provide insights and theoretical foundations for future
works of building non-compressive vision models. Code is available at
https://github.com/wangf3014/Patch_Scaling.
Reducing patch size in image tokenization consistently enhances predictive performance in Vision Transformers and Mamba models, suggesting that pixel tokenization is optimal for visual understanding.
Vision Transformer ViT patchification image tokenization compressive encoding scaling law predictive performance pixel tokenization task-specific decoder heads dense prediction Mamba models visual sequence ImageNet-1k