title: Vision Transformers Need Registers
summary: Transformers have recently emerged as a powerful tool for learning visual
  representations. In this paper, we identify and characterize artifacts in
  feature maps of both supervised and self-supervised ViT networks. The artifacts
  correspond to high-norm tokens appearing during inference primarily in
  low-informative background areas of images, that are repurposed for internal
  computations. We propose a simple yet effective solution based on providing
  additional tokens to the input sequence of the Vision Transformer to fill that
  role. We show that this solution fixes that problem entirely for both
  supervised and self-supervised models, sets a new state of the art for
  self-supervised visual models on dense visual prediction tasks, enables object
  discovery methods with larger models, and most importantly leads to smoother
  feature maps and attention maps for downstream visual processing.
paper_authors._id: 65164fecb84fb7bc6cbe98d9
paper_authors.name: Timothée Darcet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:21:44.889Z
paper_authors._id: 63047da67424d937fa31b2b0
paper_authors.avatarUrl: /avatars/cfdbb2054da0af37834a619f6c03db52.svg
paper_authors.fullname: Timothée Darcet
paper_authors.type: user
paper_authors.user: TimDarcet
paper_authors._id: 65164fecb84fb7bc6cbe98da
paper_authors.name: Maxime Oquab
paper_authors._id: 65164fecb84fb7bc6cbe98db
paper_authors.name: Julien Mairal
paper_authors._id: 65164fecb84fb7bc6cbe98dc
paper_authors.name: Piotr Bojanowski
paper_title: Vision Transformers Need Registers
paper_summary: Transformers have recently emerged as a powerful tool for learning visual
  representations. In this paper, we identify and characterize artifacts in
  feature maps of both supervised and self-supervised ViT networks. The artifacts
  correspond to high-norm tokens appearing during inference primarily in
  low-informative background areas of images, that are repurposed for internal
  computations. We propose a simple yet effective solution based on providing
  additional tokens to the input sequence of the Vision Transformer to fill that
  role. We show that this solution fixes that problem entirely for both
  supervised and self-supervised models, sets a new state of the art for
  self-supervised visual models on dense visual prediction tasks, enables object
  discovery methods with larger models, and most importantly leads to smoother
  feature maps and attention maps for downstream visual processing.
paper_ai_keywords: Transformers feature maps ViT networks high-norm tokens low-informative background areas input sequence Vision Transformer additional tokens state of the art dense visual prediction tasks object discovery methods smoother feature maps attention maps downstream visual processing