title: Vision Transformers with Self-Distilled Registers
summary: Vision Transformers (ViTs) have emerged as the dominant architecture for
  visual processing tasks, demonstrating excellent scalability with increased
  training data and model size. However, recent work has identified the emergence
  of artifact tokens in ViTs that are incongruous with the local semantics. These
  anomalous tokens degrade ViT performance in tasks that require fine-grained
  localization or structural coherence. An effective mitigation of this issue is
  to the addition of register tokens to ViTs, which implicitly "absorb" the
  artifact term during training. Given the availability of various large-scale
  pre-trained ViTs, in this paper we aim at equipping them with such register
  tokens without the need of re-training them from scratch, which is infeasible
  considering their size. Specifically, we propose Post Hoc Registers (PH-Reg),
  an efficient self-distillation method that integrates registers into an
  existing ViT without requiring additional labeled data and full retraining.
  PH-Reg initializes both teacher and student networks from the same pre-trained
  ViT. The teacher remains frozen and unmodified, while the student is augmented
  with randomly initialized register tokens. By applying test-time augmentation
  to the teacher's inputs, we generate denoised dense embeddings free of
  artifacts, which are then used to optimize only a small subset of unlocked
  student weights. We show that our approach can effectively reduce the number of
  artifact tokens, improving the segmentation and depth prediction of the student
  ViT under zero-shot and linear probing.
paper_authors._id: 68373c9bb93aa1e23ae48a27
paper_authors.name: Yinjie Chen
paper_authors._id: 68373c9bb93aa1e23ae48a28
paper_authors.name: Zipeng Yan
paper_authors._id: 68373c9bb93aa1e23ae48a29
paper_authors.name: Chong Zhou
paper_authors._id: 68373c9bb93aa1e23ae48a2a
paper_authors.name: Bo Dai
paper_authors._id: 68373c9bb93aa1e23ae48a2b
paper_authors.name: Andrew F. Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:13:35.893Z
paper_authors._id: 64b6ce23dbbd1f2cdb624d56
paper_authors.avatarUrl: /avatars/022738ce8f76fa9545b3e363d7264b53.svg
paper_authors.fullname: Andrew Luo
paper_authors.type: user
paper_authors.user: aluo-x
paper_title: Vision Transformers with Self-Distilled Registers
paper_summary: Vision Transformers (ViTs) have emerged as the dominant architecture for
  visual processing tasks, demonstrating excellent scalability with increased
  training data and model size. However, recent work has identified the emergence
  of artifact tokens in ViTs that are incongruous with the local semantics. These
  anomalous tokens degrade ViT performance in tasks that require fine-grained
  localization or structural coherence. An effective mitigation of this issue is
  to the addition of register tokens to ViTs, which implicitly "absorb" the
  artifact term during training. Given the availability of various large-scale
  pre-trained ViTs, in this paper we aim at equipping them with such register
  tokens without the need of re-training them from scratch, which is infeasible
  considering their size. Specifically, we propose Post Hoc Registers (PH-Reg),
  an efficient self-distillation method that integrates registers into an
  existing ViT without requiring additional labeled data and full retraining.
  PH-Reg initializes both teacher and student networks from the same pre-trained
  ViT. The teacher remains frozen and unmodified, while the student is augmented
  with randomly initialized register tokens. By applying test-time augmentation
  to the teacher's inputs, we generate denoised dense embeddings free of
  artifacts, which are then used to optimize only a small subset of unlocked
  student weights. We show that our approach can effectively reduce the number of
  artifact tokens, improving the segmentation and depth prediction of the student
  ViT under zero-shot and linear probing.
paper_ai_summary: Post Hoc Registers, a self-distillation method, integrates registers into pre-trained Vision Transformers to reduce artifact tokens, enhancing segmentation and depth prediction.
paper_ai_keywords: vision transformers artifact tokens register tokens self-distillation test-time augmentation denoised dense embeddings segmentation depth prediction zero-shot learning linear probing