title: SHIC: Shape-Image Correspondences with no Keypoint Supervision
summary: Canonical surface mapping generalizes keypoint detection by assigning each
  pixel of an object to a corresponding point in a 3D template. Popularised by
  DensePose for the analysis of humans, authors have since attempted to apply the
  concept to more categories, but with limited success due to the high cost of
  manual supervision. In this work, we introduce SHIC, a method to learn
  canonical maps without manual supervision which achieves better results than
  supervised methods for most categories. Our idea is to leverage foundation
  computer vision models such as DINO and Stable Diffusion that are open-ended
  and thus possess excellent priors over natural categories. SHIC reduces the
  problem of estimating image-to-template correspondences to predicting
  image-to-image correspondences using features from the foundation models. The
  reduction works by matching images of the object to non-photorealistic renders
  of the template, which emulates the process of collecting manual annotations
  for this task. These correspondences are then used to supervise high-quality
  canonical maps for any object of interest. We also show that image generators
  can further improve the realism of the template views, which provide an
  additional source of supervision for the model.
paper_authors._id: 66a705711dfa319a5ba83051
paper_authors.name: Aleksandar Shtedritski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:44:39.831Z
paper_authors._id: 5fc3a2b8ce45c15cb22bf0dc
paper_authors.avatarUrl: /avatars/9065a49ace1c4a54e3acba8e72a128e1.svg
paper_authors.fullname: Aleksandar Shtedritski
paper_authors.type: user
paper_authors.user: suny-sht
paper_authors._id: 66a705711dfa319a5ba83052
paper_authors.name: Christian Rupprecht
paper_authors._id: 66a705711dfa319a5ba83053
paper_authors.name: Andrea Vedaldi
paper_title: SHIC: Shape-Image Correspondences with no Keypoint Supervision
paper_summary: Canonical surface mapping generalizes keypoint detection by assigning each
  pixel of an object to a corresponding point in a 3D template. Popularised by
  DensePose for the analysis of humans, authors have since attempted to apply the
  concept to more categories, but with limited success due to the high cost of
  manual supervision. In this work, we introduce SHIC, a method to learn
  canonical maps without manual supervision which achieves better results than
  supervised methods for most categories. Our idea is to leverage foundation
  computer vision models such as DINO and Stable Diffusion that are open-ended
  and thus possess excellent priors over natural categories. SHIC reduces the
  problem of estimating image-to-template correspondences to predicting
  image-to-image correspondences using features from the foundation models. The
  reduction works by matching images of the object to non-photorealistic renders
  of the template, which emulates the process of collecting manual annotations
  for this task. These correspondences are then used to supervise high-quality
  canonical maps for any object of interest. We also show that image generators
  can further improve the realism of the template views, which provide an
  additional source of supervision for the model.
paper_ai_keywords: DensePose SHIC image-to-template correspondences image-to-image correspondences foundation computer vision models DINO Stable Diffusion non-photorealistic renders template views zero-shot