title: Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
summary: Since the introduction of Vision Transformer (ViT), patchification has long
  been regarded as a de facto image tokenization approach for plain visual
  architectures. By compressing the spatial size of images, this approach can
  effectively shorten the token sequence and reduce the computational cost of
  ViT-like plain architectures. In this work, we aim to thoroughly examine the
  information loss caused by this patchification-based compressive encoding
  paradigm and how it affects visual understanding. We conduct extensive patch
  size scaling experiments and excitedly observe an intriguing scaling law in
  patchification: the models can consistently benefit from decreased patch sizes
  and attain improved predictive performance, until it reaches the minimum patch
  size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable
  across different vision tasks, various input scales, and diverse architectures
  such as ViT and the recent Mamba models. Moreover, as a by-product, we discover
  that with smaller patches, task-specific decoder heads become less critical for
  dense prediction. In the experiments, we successfully scale up the visual
  sequence to an exceptional length of 50,176 tokens, achieving a competitive
  test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We
  hope this study can provide insights and theoretical foundations for future
  works of building non-compressive vision models. Code is available at
  https://github.com/wangf3014/Patch_Scaling.
paper_authors._id: 67a8d049406cb5a65f847eb1
paper_authors.name: Feng Wang
paper_authors._id: 67a8d049406cb5a65f847eb2
paper_authors.name: Yaodong Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:22:16.391Z
paper_authors._id: 6100e69a393be1b5c4c83867
paper_authors.avatarUrl: /avatars/1b87098cffb9c50345789808daea4f68.svg
paper_authors.fullname: Yaodong Yu
paper_authors.type: user
paper_authors.user: yaodongyu
paper_authors._id: 67a8d049406cb5a65f847eb3
paper_authors.name: Guoyizhe Wei
paper_authors._id: 67a8d049406cb5a65f847eb4
paper_authors.name: Wei Shao
paper_authors._id: 67a8d049406cb5a65f847eb5
paper_authors.name: Yuyin Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:22:41.354Z
paper_authors._id: 66c7fb4ce2c92fe5b132f314
paper_authors.avatarUrl: /avatars/22d915fa339a70803c5c748255250256.svg
paper_authors.fullname: Yuyin Zhou
paper_authors.type: user
paper_authors.user: RitaCoding
paper_authors._id: 67a8d049406cb5a65f847eb6
paper_authors.name: Alan Yuille
paper_authors._id: 67a8d049406cb5a65f847eb7
paper_authors.name: Cihang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:22:31.527Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_title: Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
paper_summary: Since the introduction of Vision Transformer (ViT), patchification has long
  been regarded as a de facto image tokenization approach for plain visual
  architectures. By compressing the spatial size of images, this approach can
  effectively shorten the token sequence and reduce the computational cost of
  ViT-like plain architectures. In this work, we aim to thoroughly examine the
  information loss caused by this patchification-based compressive encoding
  paradigm and how it affects visual understanding. We conduct extensive patch
  size scaling experiments and excitedly observe an intriguing scaling law in
  patchification: the models can consistently benefit from decreased patch sizes
  and attain improved predictive performance, until it reaches the minimum patch
  size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable
  across different vision tasks, various input scales, and diverse architectures
  such as ViT and the recent Mamba models. Moreover, as a by-product, we discover
  that with smaller patches, task-specific decoder heads become less critical for
  dense prediction. In the experiments, we successfully scale up the visual
  sequence to an exceptional length of 50,176 tokens, achieving a competitive
  test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We
  hope this study can provide insights and theoretical foundations for future
  works of building non-compressive vision models. Code is available at
  https://github.com/wangf3014/Patch_Scaling.
paper_ai_keywords: Vision Transformer (ViT) patchification image tokenization compressive encoding predictive performance pixel tokenization dense prediction visual sequence ImageNet-1k benchmark non-compressive vision models