title: Denoising Vision Transformers
summary: We delve into a nuanced but significant challenge inherent to Vision
  Transformers (ViTs): feature maps of these models exhibit grid-like artifacts,
  which detrimentally hurt the performance of ViTs in downstream tasks. Our
  investigations trace this fundamental issue down to the positional embeddings
  at the input stage. To address this, we propose a novel noise model, which is
  universally applicable to all ViTs. Specifically, the noise model dissects ViT
  outputs into three components: a semantics term free from noise artifacts and
  two artifact-related terms that are conditioned on pixel locations. Such a
  decomposition is achieved by enforcing cross-view feature consistency with
  neural fields in a per-image basis. This per-image optimization process
  extracts artifact-free features from raw ViT outputs, providing clean features
  for offline applications. Expanding the scope of our solution to support online
  functionality, we introduce a learnable denoiser to predict artifact-free
  features directly from unprocessed ViT outputs, which shows remarkable
  generalization capabilities to novel data without the need for per-image
  optimization. Our two-stage approach, termed Denoising Vision Transformers
  (DVT), does not require re-training existing pre-trained ViTs and is
  immediately applicable to any Transformer-based architecture. We evaluate our
  method on a variety of representative ViTs (DINO, MAE, DeiT-III, EVA02, CLIP,
  DINOv2, DINOv2-reg). Extensive evaluations demonstrate that our DVT
  consistently and significantly improves existing state-of-the-art
  general-purpose models in semantic and geometric tasks across multiple datasets
  (e.g., +3.84 mIoU). We hope our study will encourage a re-evaluation of ViT
  design, especially regarding the naive use of positional embeddings.
paper_authors._id: 659b66cba7f2d24917e1788a
paper_authors.name: Jiawei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:39:48.545Z
paper_authors._id: 6594e1079e16fa75101bda67
paper_authors.avatarUrl: /avatars/8cffecfeaef6dcec92c20fcdfbb89e8a.svg
paper_authors.fullname: Jiawei Yang
paper_authors.type: user
paper_authors.user: jjiaweiyang
paper_authors._id: 659b66cba7f2d24917e1788b
paper_authors.name: Katie Z Luo
paper_authors._id: 659b66cba7f2d24917e1788c
paper_authors.name: Jiefeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:41:20.137Z
paper_authors._id: 630b5be6a20a5367811a0a2f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664521386819-630b5be6a20a5367811a0a2f.jpeg
paper_authors.fullname: Jiefeng Li
paper_authors.type: user
paper_authors.user: JeffLi
paper_authors._id: 659b66cba7f2d24917e1788d
paper_authors.name: Kilian Q Weinberger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:41:33.983Z
paper_authors._id: 60a7f4b44e24361791fe8238
paper_authors.avatarUrl: /avatars/c33475f7740d6890733654b002a2d2ad.svg
paper_authors.fullname: Weinberger
paper_authors.type: user
paper_authors.user: Kilian
paper_authors._id: 659b66cba7f2d24917e1788e
paper_authors.name: Yonglong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:41:55.267Z
paper_authors._id: 64793086ac77bb7603842456
paper_authors.avatarUrl: /avatars/5891292138d372a9cc063f86c08b8174.svg
paper_authors.fullname: Yonglong Tian
paper_authors.type: user
paper_authors.user: tyl5566
paper_authors._id: 659b66cba7f2d24917e1788f
paper_authors.name: Yue Wang
paper_title: Denoising Vision Transformers
paper_summary: We delve into a nuanced but significant challenge inherent to Vision
  Transformers (ViTs): feature maps of these models exhibit grid-like artifacts,
  which detrimentally hurt the performance of ViTs in downstream tasks. Our
  investigations trace this fundamental issue down to the positional embeddings
  at the input stage. To address this, we propose a novel noise model, which is
  universally applicable to all ViTs. Specifically, the noise model dissects ViT
  outputs into three components: a semantics term free from noise artifacts and
  two artifact-related terms that are conditioned on pixel locations. Such a
  decomposition is achieved by enforcing cross-view feature consistency with
  neural fields in a per-image basis. This per-image optimization process
  extracts artifact-free features from raw ViT outputs, providing clean features
  for offline applications. Expanding the scope of our solution to support online
  functionality, we introduce a learnable denoiser to predict artifact-free
  features directly from unprocessed ViT outputs, which shows remarkable
  generalization capabilities to novel data without the need for per-image
  optimization. Our two-stage approach, termed Denoising Vision Transformers
  (DVT), does not require re-training existing pre-trained ViTs and is
  immediately applicable to any Transformer-based architecture. We evaluate our
  method on a variety of representative ViTs (DINO, MAE, DeiT-III, EVA02, CLIP,
  DINOv2, DINOv2-reg). Extensive evaluations demonstrate that our DVT
  consistently and significantly improves existing state-of-the-art
  general-purpose models in semantic and geometric tasks across multiple datasets
  (e.g., +3.84 mIoU). We hope our study will encourage a re-evaluation of ViT
  design, especially regarding the naive use of positional embeddings.
paper_ai_summary: A novel noise model is proposed to eliminate grid-like artifacts in Vision Transformers by decomposing outputs into noise-free and artifact-related components, enhancing performance in downstream tasks.
paper_ai_keywords: Vision Transformers ViTs feature maps grid-like artifacts positional embeddings noise model semantics term artifact-related terms cross-view feature consistency neural fields per-image optimization learnable denoiser Denoising Vision Transformers DVT DINO MAE DeiT-III EVA02 CLIP DINOv2 DINOv2-reg semantic tasks geometric tasks mIoU