Vision Transformers with Self-Distilled Registers
Vision Transformers (ViTs) have emerged as the dominant architecture for
visual processing tasks, demonstrating excellent scalability with increased
training data and model size. However, recent work has identified the emergence
of artifact tokens in ViTs that are incongruous with the local semantics. These
anomalous tokens degrade ViT performance in tasks that require fine-grained
localization or structural coherence. An effective mitigation of this issue is
to the addition of register tokens to ViTs, which implicitly "absorb" the
artifact term during training. Given the availability of various large-scale
pre-trained ViTs, in this paper we aim at equipping them with such register
tokens without the need of re-training them from scratch, which is infeasible
considering their size. Specifically, we propose Post Hoc Registers (PH-Reg),
an efficient self-distillation method that integrates registers into an
existing ViT without requiring additional labeled data and full retraining.
PH-Reg initializes both teacher and student networks from the same pre-trained
ViT. The teacher remains frozen and unmodified, while the student is augmented
with randomly initialized register tokens. By applying test-time augmentation
to the teacher's inputs, we generate denoised dense embeddings free of
artifacts, which are then used to optimize only a small subset of unlocked
student weights. We show that our approach can effectively reduce the number of
artifact tokens, improving the segmentation and depth prediction of the student
ViT under zero-shot and linear probing.
68373c9bb93aa1e23ae48a27
Yinjie Chen
68373c9bb93aa1e23ae48a28
Zipeng Yan
68373c9bb93aa1e23ae48a29
Chong Zhou
68373c9bb93aa1e23ae48a2a
Bo Dai
68373c9bb93aa1e23ae48a2b
Andrew F. Luo
claimed_verified
2025-05-28T19:13:35.893Z
64b6ce23dbbd1f2cdb624d56
/avatars/022738ce8f76fa9545b3e363d7264b53.svg
Andrew Luo
user
aluo-x
Vision Transformers with Self-Distilled Registers
Vision Transformers (ViTs) have emerged as the dominant architecture for
visual processing tasks, demonstrating excellent scalability with increased
training data and model size. However, recent work has identified the emergence
of artifact tokens in ViTs that are incongruous with the local semantics. These
anomalous tokens degrade ViT performance in tasks that require fine-grained
localization or structural coherence. An effective mitigation of this issue is
to the addition of register tokens to ViTs, which implicitly "absorb" the
artifact term during training. Given the availability of various large-scale
pre-trained ViTs, in this paper we aim at equipping them with such register
tokens without the need of re-training them from scratch, which is infeasible
considering their size. Specifically, we propose Post Hoc Registers (PH-Reg),
an efficient self-distillation method that integrates registers into an
existing ViT without requiring additional labeled data and full retraining.
PH-Reg initializes both teacher and student networks from the same pre-trained
ViT. The teacher remains frozen and unmodified, while the student is augmented
with randomly initialized register tokens. By applying test-time augmentation
to the teacher's inputs, we generate denoised dense embeddings free of
artifacts, which are then used to optimize only a small subset of unlocked
student weights. We show that our approach can effectively reduce the number of
artifact tokens, improving the segmentation and depth prediction of the student
ViT under zero-shot and linear probing.
Post Hoc Registers, a self-distillation method, integrates registers into pre-trained Vision Transformers to reduce artifact tokens, enhancing segmentation and depth prediction.
vision transformers artifact tokens register tokens self-distillation test-time augmentation denoised dense embeddings segmentation depth prediction zero-shot learning linear probing