DINeMo: Learning Neural Mesh Models with no 3D Annotations
Category-level 3D/6D pose estimation is a crucial step towards comprehensive
3D scene understanding, which would enable a broad range of applications in
robotics and embodied AI. Recent works explored neural mesh models that
approach a range of 2D and 3D tasks from an analysis-by-synthesis perspective.
Despite the largely enhanced robustness to partial occlusion and domain shifts,
these methods depended heavily on 3D annotations for part-contrastive learning,
which confines them to a narrow set of categories and hinders efficient
scaling. In this work, we present DINeMo, a novel neural mesh model that is
trained with no 3D annotations by leveraging pseudo-correspondence obtained
from large visual foundation models. We adopt a bidirectional
pseudo-correspondence generation method, which produce pseudo correspondence
utilize both local appearance features and global context information.
Experimental results on car datasets demonstrate that our DINeMo outperforms
previous zero- and few-shot 3D pose estimation by a wide margin, narrowing the
gap with fully-supervised methods by 67.3%. Our DINeMo also scales effectively
and efficiently when incorporating more unlabeled images during training, which
demonstrate the advantages over supervised learning methods that rely on 3D
annotations. Our project page is available at
https://analysis-by-synthesis.github.io/DINeMo/.
67e4b035af7d0551dc377e13
Weijie Guo
67e4b035af7d0551dc377e14
Guofeng Zhang
admin_assigned
2025-03-27T10:15:46.953Z
661434dd0dcdede49138cd7c
/avatars/35f08bf08c5e9edfb3c78e280af718cb.svg
Guofeng Zhang
user
guofeng1123
67e4b035af7d0551dc377e15
Wufei Ma
admin_assigned
2025-03-27T10:15:53.618Z
625f81afe1994410eef1c36a
https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg
Wufei Ma
user
wufeim
67e4b035af7d0551dc377e16
Alan Yuille
DINeMo: Learning Neural Mesh Models with no 3D Annotations
Category-level 3D/6D pose estimation is a crucial step towards comprehensive
3D scene understanding, which would enable a broad range of applications in
robotics and embodied AI. Recent works explored neural mesh models that
approach a range of 2D and 3D tasks from an analysis-by-synthesis perspective.
Despite the largely enhanced robustness to partial occlusion and domain shifts,
these methods depended heavily on 3D annotations for part-contrastive learning,
which confines them to a narrow set of categories and hinders efficient
scaling. In this work, we present DINeMo, a novel neural mesh model that is
trained with no 3D annotations by leveraging pseudo-correspondence obtained
from large visual foundation models. We adopt a bidirectional
pseudo-correspondence generation method, which produce pseudo correspondence
utilize both local appearance features and global context information.
Experimental results on car datasets demonstrate that our DINeMo outperforms
previous zero- and few-shot 3D pose estimation by a wide margin, narrowing the
gap with fully-supervised methods by 67.3%. Our DINeMo also scales effectively
and efficiently when incorporating more unlabeled images during training, which
demonstrate the advantages over supervised learning methods that rely on 3D
annotations. Our project page is available at
https://analysis-by-synthesis.github.io/DINeMo/.
DINeMo, a novel neural mesh model, achieves superior zero- and few-shot 3D pose estimation by generating pseudo-correspondences through large visual foundation models, outperforming supervised methods.
neural mesh models analysis-by-synthesis pseudo-correspondence visual foundation models bidirectional pseudo-correspondence 3D pose estimation 3D annotations car datasets zero-shot few-shot supervised learning