ViTAR: Vision Transformer with Any Resolution
his paper tackles a significant challenge faced by Vision Transformers
(ViTs): their constrained scalability across different image resolutions.
Typically, ViTs experience a performance decline when processing resolutions
different from those seen during training. Our work introduces two key
innovations to address this issue. Firstly, we propose a novel module for
dynamic resolution adjustment, designed with a single Transformer block,
specifically to achieve highly efficient incremental token integration.
Secondly, we introduce fuzzy positional encoding in the Vision Transformer to
provide consistent positional awareness across multiple resolutions, thereby
preventing overfitting to any single training resolution. Our resulting model,
ViTAR (Vision Transformer with Any Resolution), demonstrates impressive
adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and
80.4\% accuracy at a 4032x4032 resolution, all while reducing computational
costs. ViTAR also shows strong performance in downstream tasks such as instance
and semantic segmentation and can easily combined with self-supervised learning
techniques like Masked AutoEncoder. Our work provides a cost-effective solution
for enhancing the resolution scalability of ViTs, paving the way for more
versatile and efficient high-resolution image processing.
6604d4fbca677fbfd5216bea
Qihang Fan
6604d4fbca677fbfd5216beb
Quanzeng You
claimed_verified
2024-04-03T09:18:19.577Z
642bb8c87f152f6e72b8d3cd
https://cdn-avatars.huggingface.co/v1/production/uploads/642bb8c87f152f6e72b8d3cd/MXxTssocydbabGZ6XFIic.png
Quanzeng You
user
Ye27
6604d4fbca677fbfd5216bec
Xiaotian Han
claimed_verified
2024-03-28T16:39:45.512Z
650dde4ce14eeb01d42b37a1
https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
Xiaotian Han
user
xiaotianhan
6604d4fbca677fbfd5216bed
Yongfei Liu
6604d4fbca677fbfd5216bee
Yunzhe Tao
6604d4fbca677fbfd5216bef
Huaibo Huang
6604d4fbca677fbfd5216bf0
Ran He
6604d4fbca677fbfd5216bf1
Hongxia Yang
ViTAR: Vision Transformer with Any Resolution
his paper tackles a significant challenge faced by Vision Transformers
(ViTs): their constrained scalability across different image resolutions.
Typically, ViTs experience a performance decline when processing resolutions
different from those seen during training. Our work introduces two key
innovations to address this issue. Firstly, we propose a novel module for
dynamic resolution adjustment, designed with a single Transformer block,
specifically to achieve highly efficient incremental token integration.
Secondly, we introduce fuzzy positional encoding in the Vision Transformer to
provide consistent positional awareness across multiple resolutions, thereby
preventing overfitting to any single training resolution. Our resulting model,
ViTAR (Vision Transformer with Any Resolution), demonstrates impressive
adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and
80.4\% accuracy at a 4032x4032 resolution, all while reducing computational
costs. ViTAR also shows strong performance in downstream tasks such as instance
and semantic segmentation and can easily combined with self-supervised learning
techniques like Masked AutoEncoder. Our work provides a cost-effective solution
for enhancing the resolution scalability of ViTs, paving the way for more
versatile and efficient high-resolution image processing.
ViTAR enhances Vision Transformers' scalability across resolutions through dynamic token integration and fuzzy positional encoding, improving accuracy and reducing computational costs.
Vision Transformers dynamic resolution adjustment incremental token integration fuzzy positional encoding ViTAR top-1 accuracy instance segmentation semantic segmentation Masked AutoEncoder