MIMIC: Masked Image Modeling with Image Correspondences
Many pixelwise dense prediction tasks-depth estimation and semantic
segmentation in computer vision today rely on pretrained image representations.
Therefore, curating effective pretraining datasets is vital. Unfortunately, the
effective pretraining datasets are those with multi-view scenes and have only
been curated using annotated 3D meshes, point clouds, and camera parameters
from simulated environments. We propose a dataset-curation mechanism that does
not require any annotations. We mine two datasets: MIMIC-1M with 1.3M and
MIMIC-3M with 3.1M multi-view image pairs from open-sourced video datasets and
from synthetic 3D environments. We train multiple self-supervised models with
different masked image modeling objectives to showcase the following findings:
Representations trained on MIMIC-3M outperform those mined using annotations on
multiple downstream tasks, including depth estimation, semantic segmentation,
surface normals, and pose estimation. They also outperform representations that
are frozen and when downstream training data is limited to few-shot. Larger
dataset (MIMIC-3M) significantly improves performance, which is promising since
our curation method can arbitrarily scale to produce even larger datasets.
MIMIC code, dataset, and pretrained models are open-sourced at
https://github.com/RAIVNLab/MIMIC.
649b9b9cc551eb1992d85306
Kalyani Marathe
admin_assigned
2023-06-28T08:07:51.198Z
649be93e6985c787d4946e87
/avatars/0f23f8a3c644c81c84aa11786b2f358e.svg
Kalyani Marathe
user
kalyani7195
649b9b9cc551eb1992d85307
Mahtab Bigverdi
claimed_verified
2023-06-28T07:41:03.867Z
648380d1f9256e215de1ff88
/avatars/a609c74911ed4f96843debdf2e7568ca.svg
Mahtab Bigverdi
user
MahtabBg
649b9b9cc551eb1992d85308
Nishat Khan
649b9b9cc551eb1992d85309
Tuhin Kundu
claimed_verified
2023-08-02T07:40:20.396Z
64b8491203124195cd795cad
/avatars/75b0252a17af0de0bd6727f3290577f9.svg
Tuhin Kundu
user
tuhink
649b9b9cc551eb1992d8530a
Aniruddha Kembhavi
649b9b9cc551eb1992d8530b
Linda G. Shapiro
649b9b9cc551eb1992d8530c
Ranjay Krishna
MIMIC: Masked Image Modeling with Image Correspondences
Many pixelwise dense prediction tasks-depth estimation and semantic
segmentation in computer vision today rely on pretrained image representations.
Therefore, curating effective pretraining datasets is vital. Unfortunately, the
effective pretraining datasets are those with multi-view scenes and have only
been curated using annotated 3D meshes, point clouds, and camera parameters
from simulated environments. We propose a dataset-curation mechanism that does
not require any annotations. We mine two datasets: MIMIC-1M with 1.3M and
MIMIC-3M with 3.1M multi-view image pairs from open-sourced video datasets and
from synthetic 3D environments. We train multiple self-supervised models with
different masked image modeling objectives to showcase the following findings:
Representations trained on MIMIC-3M outperform those mined using annotations on
multiple downstream tasks, including depth estimation, semantic segmentation,
surface normals, and pose estimation. They also outperform representations that
are frozen and when downstream training data is limited to few-shot. Larger
dataset (MIMIC-3M) significantly improves performance, which is promising since
our curation method can arbitrarily scale to produce even larger datasets.
MIMIC code, dataset, and pretrained models are open-sourced at
https://github.com/RAIVNLab/MIMIC.
A dataset-curation method for pixelwise dense prediction tasks, using unlabeled video datasets and synthetic environments, produces larger datasets that outperform annotated ones and enhance downstream task performance.
self-supervised models masked image modeling depth estimation semantic segmentation surface normals pose estimation few-shot learning