title: NeCo: Improving DINOv2's spatial representations in 19 GPU hours with
    Patch Neighbor Consistency
summary: We propose sorting patch representations across views as a novel
  self-supervised learning signal to improve pretrained representations. To this
  end, we introduce NeCo: Patch Neighbor Consistency, a novel training loss that
  enforces patch-level nearest neighbor consistency across a student and teacher
  model, relative to reference batches. Our method leverages a differentiable
  sorting method applied on top of pretrained representations, such as
  DINOv2-registers to bootstrap the learning signal and further improve upon
  them. This dense post-pretraining leads to superior performance across various
  models and datasets, despite requiring only 19 hours on a single GPU. We
  demonstrate that this method generates high-quality dense feature encoders and
  establish several new state-of-the-art results: +5.5% and + 6% for
  non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and
  +7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.
paper_authors._id: 66c5b3eb2d2fcb957116b49e
paper_authors.name: Valentinos Pariza
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:44:06.788Z
paper_authors._id: 63f79b1bfd6949669625f54e
paper_authors.avatarUrl: /avatars/04f7a3a9e71a656ea64aed1bb01db5b1.svg
paper_authors.fullname: Valentinos Pariza
paper_authors.type: user
paper_authors.user: valentinospariza
paper_authors._id: 66c5b3eb2d2fcb957116b49f
paper_authors.name: Mohammadreza Salehi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T08:57:40.286Z
paper_authors._id: 66990ec3aa500cd99d640a19
paper_authors.avatarUrl: /avatars/3520cab2b946158b2cdf6ed84f9516e0.svg
paper_authors.fullname: Mohammadreza Salehi
paper_authors.type: user
paper_authors.user: SMSD75
paper_authors._id: 66c5b3eb2d2fcb957116b4a0
paper_authors.name: Gertjan Burghouts
paper_authors._id: 66c5b3eb2d2fcb957116b4a1
paper_authors.name: Francesco Locatello
paper_authors._id: 66c5b3eb2d2fcb957116b4a2
paper_authors.name: Yuki M. Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:34.944Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_title: NeCo: Improving DINOv2's spatial representations in 19 GPU hours with
    Patch Neighbor Consistency
paper_summary: We propose sorting patch representations across views as a novel
  self-supervised learning signal to improve pretrained representations. To this
  end, we introduce NeCo: Patch Neighbor Consistency, a novel training loss that
  enforces patch-level nearest neighbor consistency across a student and teacher
  model, relative to reference batches. Our method leverages a differentiable
  sorting method applied on top of pretrained representations, such as
  DINOv2-registers to bootstrap the learning signal and further improve upon
  them. This dense post-pretraining leads to superior performance across various
  models and datasets, despite requiring only 19 hours on a single GPU. We
  demonstrate that this method generates high-quality dense feature encoders and
  establish several new state-of-the-art results: +5.5% and + 6% for
  non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and
  +7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.
paper_ai_keywords: self-supervised learning NeCo: Patch Neighbor Consistency training loss patch-level nearest neighbor consistency student and teacher model reference batches differentiable sorting DINOv2-registers dense post-pretraining semantic segmentation ADE20k Pascal VOC linear segmentation evaluations COCO-Things COCO-Stuff