title: MIMIC: Masked Image Modeling with Image Correspondences
summary: Many pixelwise dense prediction tasks-depth estimation and semantic
  segmentation in computer vision today rely on pretrained image representations.
  Therefore, curating effective pretraining datasets is vital. Unfortunately, the
  effective pretraining datasets are those with multi-view scenes and have only
  been curated using annotated 3D meshes, point clouds, and camera parameters
  from simulated environments. We propose a dataset-curation mechanism that does
  not require any annotations. We mine two datasets: MIMIC-1M with 1.3M and
  MIMIC-3M with 3.1M multi-view image pairs from open-sourced video datasets and
  from synthetic 3D environments. We train multiple self-supervised models with
  different masked image modeling objectives to showcase the following findings:
  Representations trained on MIMIC-3M outperform those mined using annotations on
  multiple downstream tasks, including depth estimation, semantic segmentation,
  surface normals, and pose estimation. They also outperform representations that
  are frozen and when downstream training data is limited to few-shot. Larger
  dataset (MIMIC-3M) significantly improves performance, which is promising since
  our curation method can arbitrarily scale to produce even larger datasets.
  MIMIC code, dataset, and pretrained models are open-sourced at
  https://github.com/RAIVNLab/MIMIC.
paper_authors._id: 649b9b9cc551eb1992d85306
paper_authors.name: Kalyani Marathe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-28T08:07:51.198Z
paper_authors._id: 649be93e6985c787d4946e87
paper_authors.avatarUrl: /avatars/0f23f8a3c644c81c84aa11786b2f358e.svg
paper_authors.fullname: Kalyani Marathe
paper_authors.type: user
paper_authors.user: kalyani7195
paper_authors._id: 649b9b9cc551eb1992d85307
paper_authors.name: Mahtab Bigverdi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T07:41:03.867Z
paper_authors._id: 648380d1f9256e215de1ff88
paper_authors.avatarUrl: /avatars/a609c74911ed4f96843debdf2e7568ca.svg
paper_authors.fullname: Mahtab Bigverdi
paper_authors.type: user
paper_authors.user: MahtabBg
paper_authors._id: 649b9b9cc551eb1992d85308
paper_authors.name: Nishat Khan
paper_authors._id: 649b9b9cc551eb1992d85309
paper_authors.name: Tuhin Kundu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-02T07:40:20.396Z
paper_authors._id: 64b8491203124195cd795cad
paper_authors.avatarUrl: /avatars/75b0252a17af0de0bd6727f3290577f9.svg
paper_authors.fullname: Tuhin Kundu
paper_authors.type: user
paper_authors.user: tuhink
paper_authors._id: 649b9b9cc551eb1992d8530a
paper_authors.name: Aniruddha Kembhavi
paper_authors._id: 649b9b9cc551eb1992d8530b
paper_authors.name: Linda G. Shapiro
paper_authors._id: 649b9b9cc551eb1992d8530c
paper_authors.name: Ranjay Krishna
paper_title: MIMIC: Masked Image Modeling with Image Correspondences
paper_summary: Many pixelwise dense prediction tasks-depth estimation and semantic
  segmentation in computer vision today rely on pretrained image representations.
  Therefore, curating effective pretraining datasets is vital. Unfortunately, the
  effective pretraining datasets are those with multi-view scenes and have only
  been curated using annotated 3D meshes, point clouds, and camera parameters
  from simulated environments. We propose a dataset-curation mechanism that does
  not require any annotations. We mine two datasets: MIMIC-1M with 1.3M and
  MIMIC-3M with 3.1M multi-view image pairs from open-sourced video datasets and
  from synthetic 3D environments. We train multiple self-supervised models with
  different masked image modeling objectives to showcase the following findings:
  Representations trained on MIMIC-3M outperform those mined using annotations on
  multiple downstream tasks, including depth estimation, semantic segmentation,
  surface normals, and pose estimation. They also outperform representations that
  are frozen and when downstream training data is limited to few-shot. Larger
  dataset (MIMIC-3M) significantly improves performance, which is promising since
  our curation method can arbitrarily scale to produce even larger datasets.
  MIMIC code, dataset, and pretrained models are open-sourced at
  https://github.com/RAIVNLab/MIMIC.
paper_ai_summary: A dataset-curation method for pixelwise dense prediction tasks, using unlabeled video datasets and synthetic environments, produces larger datasets that outperform annotated ones and enhance downstream task performance.
paper_ai_keywords: self-supervised models masked image modeling depth estimation semantic segmentation surface normals pose estimation few-shot learning