SHIC: Shape-Image Correspondences with no Keypoint Supervision
Canonical surface mapping generalizes keypoint detection by assigning each
pixel of an object to a corresponding point in a 3D template. Popularised by
DensePose for the analysis of humans, authors have since attempted to apply the
concept to more categories, but with limited success due to the high cost of
manual supervision. In this work, we introduce SHIC, a method to learn
canonical maps without manual supervision which achieves better results than
supervised methods for most categories. Our idea is to leverage foundation
computer vision models such as DINO and Stable Diffusion that are open-ended
and thus possess excellent priors over natural categories. SHIC reduces the
problem of estimating image-to-template correspondences to predicting
image-to-image correspondences using features from the foundation models. The
reduction works by matching images of the object to non-photorealistic renders
of the template, which emulates the process of collecting manual annotations
for this task. These correspondences are then used to supervise high-quality
canonical maps for any object of interest. We also show that image generators
can further improve the realism of the template views, which provide an
additional source of supervision for the model.
66a705711dfa319a5ba83051
Aleksandar Shtedritski
admin_assigned
2024-07-29T08:44:39.831Z
5fc3a2b8ce45c15cb22bf0dc
/avatars/9065a49ace1c4a54e3acba8e72a128e1.svg
Aleksandar Shtedritski
user
suny-sht
66a705711dfa319a5ba83052
Christian Rupprecht
66a705711dfa319a5ba83053
Andrea Vedaldi
SHIC: Shape-Image Correspondences with no Keypoint Supervision
Canonical surface mapping generalizes keypoint detection by assigning each
pixel of an object to a corresponding point in a 3D template. Popularised by
DensePose for the analysis of humans, authors have since attempted to apply the
concept to more categories, but with limited success due to the high cost of
manual supervision. In this work, we introduce SHIC, a method to learn
canonical maps without manual supervision which achieves better results than
supervised methods for most categories. Our idea is to leverage foundation
computer vision models such as DINO and Stable Diffusion that are open-ended
and thus possess excellent priors over natural categories. SHIC reduces the
problem of estimating image-to-template correspondences to predicting
image-to-image correspondences using features from the foundation models. The
reduction works by matching images of the object to non-photorealistic renders
of the template, which emulates the process of collecting manual annotations
for this task. These correspondences are then used to supervise high-quality
canonical maps for any object of interest. We also show that image generators
can further improve the realism of the template views, which provide an
additional source of supervision for the model.
SHIC leverages foundation computer vision models to learn canonical surface maps without manual supervision, achieving superior results by simulating the annotation process using image-to-image correspondences and enhanced template views.
DensePose keypoint detection canonical surface mapping SHIC DINO Stable Diffusion image-to-template correspondences image-to-image correspondences non-photorealistic renders image generators