title: MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
summary: The input tokens to Vision Transformers carry little semantic meaning as they
  are defined as regular equal-sized patches of the input image, regardless of
  its content. However, processing uniform background areas of an image should
  not necessitate as much compute as dense, cluttered areas. To address this
  issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our
  method introduces a conditional gating mechanism that selects the optimal token
  scale for every image region, such that the number of tokens is dynamically
  determined per input. The proposed gating module is lightweight, agnostic to
  the choice of transformer backbone, and trained within a few epochs (e.g., 20
  epochs on ImageNet) with little training overhead. In addition, to enhance the
  conditional behavior of the gate during training, we introduce a novel
  generalization of the batch-shaping loss. We show that our gating module is
  able to learn meaningful semantics despite operating locally at the coarse
  patch-level. We validate MSViT on the tasks of classification and segmentation
  where it leads to improved accuracy-complexity trade-off.
paper_authors._id: 64a6456a01c586fa61330fd0
paper_authors.name: Jakob Drachmann Havtorn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T07:34:53.837Z
paper_authors._id: 64a6dd620517081ff06a6930
paper_authors.avatarUrl: /avatars/521c0346cfe877a6c0b193c8a6490eba.svg
paper_authors.fullname: Jakob Drachmann Havtorn
paper_authors.type: user
paper_authors.user: Jaksen93
paper_authors._id: 64a6456a01c586fa61330fd1
paper_authors.name: Amelie Royer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-06T08:53:49.378Z
paper_authors._id: 66b1ded53ddfb745eddfd072
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JD85tZKReU8CWS-QDWnTU.png
paper_authors.fullname: Amelie Royer
paper_authors.type: user
paper_authors.user: ameroyer
paper_authors._id: 64a6456a01c586fa61330fd2
paper_authors.name: Tijmen Blankevoort
paper_authors._id: 64a6456a01c586fa61330fd3
paper_authors.name: Babak Ehteshami Bejnordi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T07:35:00.615Z
paper_authors._id: 645ce510f5760d1530d322cb
paper_authors.avatarUrl: /avatars/139d1b5b2e6a83ef39298e4dc17258ff.svg
paper_authors.fullname: Babak Ehteshami Bejnordi
paper_authors.type: user
paper_authors.user: babakint
paper_title: MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
paper_summary: The input tokens to Vision Transformers carry little semantic meaning as they
  are defined as regular equal-sized patches of the input image, regardless of
  its content. However, processing uniform background areas of an image should
  not necessitate as much compute as dense, cluttered areas. To address this
  issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our
  method introduces a conditional gating mechanism that selects the optimal token
  scale for every image region, such that the number of tokens is dynamically
  determined per input. The proposed gating module is lightweight, agnostic to
  the choice of transformer backbone, and trained within a few epochs (e.g., 20
  epochs on ImageNet) with little training overhead. In addition, to enhance the
  conditional behavior of the gate during training, we introduce a novel
  generalization of the batch-shaping loss. We show that our gating module is
  able to learn meaningful semantics despite operating locally at the coarse
  patch-level. We validate MSViT on the tasks of classification and segmentation
  where it leads to improved accuracy-complexity trade-off.
paper_ai_summary: A dynamic mixed-scale tokenization scheme for Vision Transformers adjusts token scale based on image content, improving accuracy and computational efficiency.
paper_ai_keywords: Vision Transformers tokenization scheme MSViT conditional gating mechanism batch-shaping loss classification segmentation accuracy-complexity trade-off