Vision Transformers Need Registers
Transformers have recently emerged as a powerful tool for learning visual
representations. In this paper, we identify and characterize artifacts in
feature maps of both supervised and self-supervised ViT networks. The artifacts
correspond to high-norm tokens appearing during inference primarily in
low-informative background areas of images, that are repurposed for internal
computations. We propose a simple yet effective solution based on providing
additional tokens to the input sequence of the Vision Transformer to fill that
role. We show that this solution fixes that problem entirely for both
supervised and self-supervised models, sets a new state of the art for
self-supervised visual models on dense visual prediction tasks, enables object
discovery methods with larger models, and most importantly leads to smoother
feature maps and attention maps for downstream visual processing.
65164fecb84fb7bc6cbe98d9
Timothée Darcet
admin_assigned
2023-09-29T11:21:44.889Z
63047da67424d937fa31b2b0
/avatars/cfdbb2054da0af37834a619f6c03db52.svg
Timothée Darcet
user
TimDarcet
65164fecb84fb7bc6cbe98da
Maxime Oquab
65164fecb84fb7bc6cbe98db
Julien Mairal
65164fecb84fb7bc6cbe98dc
Piotr Bojanowski
Vision Transformers Need Registers
Transformers have recently emerged as a powerful tool for learning visual
representations. In this paper, we identify and characterize artifacts in
feature maps of both supervised and self-supervised ViT networks. The artifacts
correspond to high-norm tokens appearing during inference primarily in
low-informative background areas of images, that are repurposed for internal
computations. We propose a simple yet effective solution based on providing
additional tokens to the input sequence of the Vision Transformer to fill that
role. We show that this solution fixes that problem entirely for both
supervised and self-supervised models, sets a new state of the art for
self-supervised visual models on dense visual prediction tasks, enables object
discovery methods with larger models, and most importantly leads to smoother
feature maps and attention maps for downstream visual processing.
Additional input tokens in Vision Transformers mitigate artifacts in feature maps, enhancing performance and enabling smoother visual processing.
Transformers feature maps ViT networks high-norm tokens self-supervised object discovery attention maps dense visual prediction tasks