title: 3D Congealing: 3D-Aware Image Alignment in the Wild
summary: We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images
  capturing semantically similar objects. Given a collection of unlabeled
  Internet images, our goal is to associate the shared semantic parts from the
  inputs and aggregate the knowledge from 2D images to a shared 3D canonical
  space. We introduce a general framework that tackles the task without assuming
  shape templates, poses, or any camera parameters. At its core is a canonical 3D
  representation that encapsulates geometric and semantic information. The
  framework optimizes for the canonical representation together with the pose for
  each input image, and a per-image coordinate map that warps 2D pixel
  coordinates to the 3D canonical frame to account for the shape matching. The
  optimization procedure fuses prior knowledge from a pre-trained image
  generative model and semantic information from input images. The former
  provides strong knowledge guidance for this under-constraint task, while the
  latter provides the necessary information to mitigate the training data bias
  from the pre-trained model. Our framework can be used for various tasks such as
  correspondence matching, pose estimation, and image editing, achieving strong
  results on real-world image datasets under challenging illumination conditions
  and on in-the-wild online image collections.
paper_authors._id: 660ce483c41c8804fbe43d66
paper_authors.name: Yunzhi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-04T07:50:28.331Z
paper_authors._id: 63793aee2cb50dda39d6e73b
paper_authors.avatarUrl: /avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
paper_authors.fullname: Yunzhi Zhang
paper_authors.type: user
paper_authors.user: Yunzhi
paper_authors._id: 660ce483c41c8804fbe43d67
paper_authors.name: Zizhang Li
paper_authors._id: 660ce483c41c8804fbe43d68
paper_authors.name: Amit Raj
paper_authors._id: 660ce483c41c8804fbe43d69
paper_authors.name: Andreas Engelhardt
paper_authors._id: 660ce483c41c8804fbe43d6a
paper_authors.name: Yuanzhen Li
paper_authors._id: 660ce483c41c8804fbe43d6b
paper_authors.name: Tingbo Hou
paper_authors._id: 660ce483c41c8804fbe43d6c
paper_authors.name: Jiajun Wu
paper_authors._id: 660ce483c41c8804fbe43d6d
paper_authors.name: Varun Jampani
paper_title: 3D Congealing: 3D-Aware Image Alignment in the Wild
paper_summary: We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images
  capturing semantically similar objects. Given a collection of unlabeled
  Internet images, our goal is to associate the shared semantic parts from the
  inputs and aggregate the knowledge from 2D images to a shared 3D canonical
  space. We introduce a general framework that tackles the task without assuming
  shape templates, poses, or any camera parameters. At its core is a canonical 3D
  representation that encapsulates geometric and semantic information. The
  framework optimizes for the canonical representation together with the pose for
  each input image, and a per-image coordinate map that warps 2D pixel
  coordinates to the 3D canonical frame to account for the shape matching. The
  optimization procedure fuses prior knowledge from a pre-trained image
  generative model and semantic information from input images. The former
  provides strong knowledge guidance for this under-constraint task, while the
  latter provides the necessary information to mitigate the training data bias
  from the pre-trained model. Our framework can be used for various tasks such as
  correspondence matching, pose estimation, and image editing, achieving strong
  results on real-world image datasets under challenging illumination conditions
  and on in-the-wild online image collections.
paper_ai_keywords: 3D Congealing 3D-aware alignment 2D images shared semantic parts 3D canonical space canonical 3D representation geometric information semantic information optimization procedure pre-trained generative model prior knowledge correspondence matching pose estimation image editing real-world image datasets challenging illumination conditions in-the-wild online image collections