Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos
<|reference_start|>Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos: Category-level 3D pose estimation is a fundamentally important problem in computer vision and robotics, e.g. for embodied agents or to train 3D generative models. However, so far methods that estimate the category-level object pose require either large amounts of human annotations, CAD models or input from RGB-D sensors. In contrast, we tackle the problem of learning to estimate the category-level 3D pose only from casually taken object-centric videos without human supervision. We propose a two-step pipeline: First, we introduce a multi-view alignment procedure that determines canonical camera poses across videos with a novel and robust cyclic distance formulation for geometric and appearance matching using reconstructed coarse meshes and DINOv2 features. In a second step, the canonical poses and reconstructed meshes enable us to train a model for 3D pose estimation from a single image. In particular, our model learns to estimate dense correspondences between images and a prototypical 3D template by predicting, for each pixel in a 2D image, a feature vector of the corresponding vertex in the template mesh. We demonstrate that our method outperforms all baselines at the unsupervised alignment of object-centric videos by a large margin and provides faithful and robust predictions in-the-wild. Our code and data is available at https://github.com/GenIntel/uns-obj-pose3d.<|reference_end|>
@article{sommer2024unsupervised,
    title={Unsupervised Learning of Category-Level 3D Pose from Object-Centric
  Videos},
    author={Leonhard Sommer, Artur Jesslen, Eddy Ilg, Adam Kortylewski},
    journal={Conference on Computer Vision and Pattern Recognition (CVPR),
  2024, pp. 22787-22796},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.04384},
    primaryClass={cs.CV}
}