title: Cluster and Predict Latents Patches for Improved Masked Image Modeling
summary: Masked Image Modeling (MIM) offers a promising approach to self-supervised
  representation learning, however existing MIM models still lag behind the
  state-of-the-art. In this paper, we systematically analyze target
  representations, loss functions, and architectures, to introduce CAPI - a novel
  pure-MIM framework that relies on the prediction of latent clusterings. Our
  approach leverages a clustering-based loss, which is stable to train, and
  exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
  accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
  substantially outperforming previous MIM methods and approaching the
  performance of the current state-of-the-art, DINOv2. We release all our code
  and models.
paper_authors._id: 67b32a554d60b7d162dffd89
paper_authors.name: Timothée Darcet
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T11:15:21.470Z
paper_authors._id: 63047da67424d937fa31b2b0
paper_authors.avatarUrl: /avatars/cfdbb2054da0af37834a619f6c03db52.svg
paper_authors.fullname: Timothée Darcet
paper_authors.type: user
paper_authors.user: TimDarcet
paper_authors._id: 67b32a554d60b7d162dffd8a
paper_authors.name: Federico Baldassarre
paper_authors._id: 67b32a554d60b7d162dffd8b
paper_authors.name: Maxime Oquab
paper_authors._id: 67b32a554d60b7d162dffd8c
paper_authors.name: Julien Mairal
paper_authors._id: 67b32a554d60b7d162dffd8d
paper_authors.name: Piotr Bojanowski
paper_title: Cluster and Predict Latents Patches for Improved Masked Image Modeling
paper_summary: Masked Image Modeling (MIM) offers a promising approach to self-supervised
  representation learning, however existing MIM models still lag behind the
  state-of-the-art. In this paper, we systematically analyze target
  representations, loss functions, and architectures, to introduce CAPI - a novel
  pure-MIM framework that relies on the prediction of latent clusterings. Our
  approach leverages a clustering-based loss, which is stable to train, and
  exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
  accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
  substantially outperforming previous MIM methods and approaching the
  performance of the current state-of-the-art, DINOv2. We release all our code
  and models.
paper_ai_keywords: Masked Image Modeling (MIM) self-supervised representation learning target representations loss functions architectures pure-MIM framework latent clusterings clustering-based loss scaling properties ViT-L backbone ImageNet mIoU ADE20K linear probes DINOv2