3D Congealing: 3D-Aware Image Alignment in the Wild
We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images
capturing semantically similar objects. Given a collection of unlabeled
Internet images, our goal is to associate the shared semantic parts from the
inputs and aggregate the knowledge from 2D images to a shared 3D canonical
space. We introduce a general framework that tackles the task without assuming
shape templates, poses, or any camera parameters. At its core is a canonical 3D
representation that encapsulates geometric and semantic information. The
framework optimizes for the canonical representation together with the pose for
each input image, and a per-image coordinate map that warps 2D pixel
coordinates to the 3D canonical frame to account for the shape matching. The
optimization procedure fuses prior knowledge from a pre-trained image
generative model and semantic information from input images. The former
provides strong knowledge guidance for this under-constraint task, while the
latter provides the necessary information to mitigate the training data bias
from the pre-trained model. Our framework can be used for various tasks such as
correspondence matching, pose estimation, and image editing, achieving strong
results on real-world image datasets under challenging illumination conditions
and on in-the-wild online image collections.
660ce483c41c8804fbe43d66
Yunzhi Zhang
claimed_verified
2024-04-04T07:50:28.331Z
63793aee2cb50dda39d6e73b
/avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
Yunzhi Zhang
user
Yunzhi
660ce483c41c8804fbe43d67
Zizhang Li
660ce483c41c8804fbe43d68
Amit Raj
660ce483c41c8804fbe43d69
Andreas Engelhardt
660ce483c41c8804fbe43d6a
Yuanzhen Li
660ce483c41c8804fbe43d6b
Tingbo Hou
660ce483c41c8804fbe43d6c
Jiajun Wu
660ce483c41c8804fbe43d6d
Varun Jampani
3D Congealing: 3D-Aware Image Alignment in the Wild
We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images
capturing semantically similar objects. Given a collection of unlabeled
Internet images, our goal is to associate the shared semantic parts from the
inputs and aggregate the knowledge from 2D images to a shared 3D canonical
space. We introduce a general framework that tackles the task without assuming
shape templates, poses, or any camera parameters. At its core is a canonical 3D
representation that encapsulates geometric and semantic information. The
framework optimizes for the canonical representation together with the pose for
each input image, and a per-image coordinate map that warps 2D pixel
coordinates to the 3D canonical frame to account for the shape matching. The
optimization procedure fuses prior knowledge from a pre-trained image
generative model and semantic information from input images. The former
provides strong knowledge guidance for this under-constraint task, while the
latter provides the necessary information to mitigate the training data bias
from the pre-trained model. Our framework can be used for various tasks such as
correspondence matching, pose estimation, and image editing, achieving strong
results on real-world image datasets under challenging illumination conditions
and on in-the-wild online image collections.
A 3D Congealing framework aligns 2D images via a shared 3D canonical space, optimizing for geometry and semantics without pre-assumed shapes or camera parameters.
3D Congealing canonical 3D representation pose estimation shape matching pre-trained image generative model semantic information correspondence matching image editing in-the-wild online image collections