title: ViTAR: Vision Transformer with Any Resolution
summary: his paper tackles a significant challenge faced by Vision Transformers
  (ViTs): their constrained scalability across different image resolutions.
  Typically, ViTs experience a performance decline when processing resolutions
  different from those seen during training. Our work introduces two key
  innovations to address this issue. Firstly, we propose a novel module for
  dynamic resolution adjustment, designed with a single Transformer block,
  specifically to achieve highly efficient incremental token integration.
  Secondly, we introduce fuzzy positional encoding in the Vision Transformer to
  provide consistent positional awareness across multiple resolutions, thereby
  preventing overfitting to any single training resolution. Our resulting model,
  ViTAR (Vision Transformer with Any Resolution), demonstrates impressive
  adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and
  80.4\% accuracy at a 4032x4032 resolution, all while reducing computational
  costs. ViTAR also shows strong performance in downstream tasks such as instance
  and semantic segmentation and can easily combined with self-supervised learning
  techniques like Masked AutoEncoder. Our work provides a cost-effective solution
  for enhancing the resolution scalability of ViTs, paving the way for more
  versatile and efficient high-resolution image processing.
paper_authors._id: 6604d4fbca677fbfd5216bea
paper_authors.name: Qihang Fan
paper_authors._id: 6604d4fbca677fbfd5216beb
paper_authors.name: Quanzeng You
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:18:19.577Z
paper_authors._id: 642bb8c87f152f6e72b8d3cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642bb8c87f152f6e72b8d3cd/MXxTssocydbabGZ6XFIic.png
paper_authors.fullname: Quanzeng You
paper_authors.type: user
paper_authors.user: Ye27
paper_authors._id: 6604d4fbca677fbfd5216bec
paper_authors.name: Xiaotian Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-28T16:39:45.512Z
paper_authors._id: 650dde4ce14eeb01d42b37a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
paper_authors.fullname: Xiaotian Han
paper_authors.type: user
paper_authors.user: xiaotianhan
paper_authors._id: 6604d4fbca677fbfd5216bed
paper_authors.name: Yongfei Liu
paper_authors._id: 6604d4fbca677fbfd5216bee
paper_authors.name: Yunzhe Tao
paper_authors._id: 6604d4fbca677fbfd5216bef
paper_authors.name: Huaibo Huang
paper_authors._id: 6604d4fbca677fbfd5216bf0
paper_authors.name: Ran He
paper_authors._id: 6604d4fbca677fbfd5216bf1
paper_authors.name: Hongxia Yang
paper_title: ViTAR: Vision Transformer with Any Resolution
paper_summary: his paper tackles a significant challenge faced by Vision Transformers
  (ViTs): their constrained scalability across different image resolutions.
  Typically, ViTs experience a performance decline when processing resolutions
  different from those seen during training. Our work introduces two key
  innovations to address this issue. Firstly, we propose a novel module for
  dynamic resolution adjustment, designed with a single Transformer block,
  specifically to achieve highly efficient incremental token integration.
  Secondly, we introduce fuzzy positional encoding in the Vision Transformer to
  provide consistent positional awareness across multiple resolutions, thereby
  preventing overfitting to any single training resolution. Our resulting model,
  ViTAR (Vision Transformer with Any Resolution), demonstrates impressive
  adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and
  80.4\% accuracy at a 4032x4032 resolution, all while reducing computational
  costs. ViTAR also shows strong performance in downstream tasks such as instance
  and semantic segmentation and can easily combined with self-supervised learning
  techniques like Masked AutoEncoder. Our work provides a cost-effective solution
  for enhancing the resolution scalability of ViTs, paving the way for more
  versatile and efficient high-resolution image processing.
paper_ai_keywords: Vision Transformer (ViT) dynamic resolution adjustment incremental token integration fuzzy positional encoding ViTAR (Vision Transformer with Any Resolution) top-1 accuracy instance segmentation semantic segmentation self-supervised learning Masked AutoEncoder resolution scalability