ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image
  Collections
Estimating 3D articulated shapes like animal bodies from monocular images is
inherently challenging due to the ambiguities of camera viewpoint, pose,
texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to
reconstruct per-instance 3D shapes from a sparse image collection in-the-wild.
Specifically, ARTIC3D is built upon a skeleton-based surface representation and
is further guided by 2D diffusion priors from Stable Diffusion. First, we
enhance the input images with occlusions/truncation via 2D diffusion to obtain
cleaner mask estimates and semantic features. Second, we perform
diffusion-guided 3D optimization to estimate shape and texture that are of
high-fidelity and faithful to input images. We also propose a novel technique
to calculate more stable image-level gradients via diffusion models compared to
existing alternatives. Finally, we produce realistic animations by fine-tuning
the rendered shape and texture under rigid part transformations. Extensive
evaluations on multiple existing datasets as well as newly introduced noisy web
image collections with occlusions and truncation demonstrate that ARTIC3D
outputs are more robust to noisy images, higher quality in terms of shape and
texture details, and more realistic when animated. Project page:
https://chhankyao.github.io/artic3d/
648145c5de559d48dbb551ec
Chun-Han Yao
admin_assigned
2023-06-12T13:50:25.340Z
63d9b9c9aefaa73525854853
/avatars/166b59f9cdd0998b354d013528c01add.svg
Chun-Han Yao
user
cyao6
648145c5de559d48dbb551ed
Amit Raj
648145c5de559d48dbb551ee
Wei-Chih Hung
648145c5de559d48dbb551ef
Yuanzhen Li
admin_assigned
2023-06-12T13:50:59.654Z
6334ee5831a2be3938c85639
/avatars/9821af26a585f54b595ec39f061f2250.svg
Yuanzhen Li
user
yzli
648145c5de559d48dbb551f0
Michael Rubinstein
648145c5de559d48dbb551f1
Ming-Hsuan Yang
648145c5de559d48dbb551f2
Varun Jampani
admin_assigned
2023-06-12T13:51:19.349Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image
  Collections
Estimating 3D articulated shapes like animal bodies from monocular images is
inherently challenging due to the ambiguities of camera viewpoint, pose,
texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to
reconstruct per-instance 3D shapes from a sparse image collection in-the-wild.
Specifically, ARTIC3D is built upon a skeleton-based surface representation and
is further guided by 2D diffusion priors from Stable Diffusion. First, we
enhance the input images with occlusions/truncation via 2D diffusion to obtain
cleaner mask estimates and semantic features. Second, we perform
diffusion-guided 3D optimization to estimate shape and texture that are of
high-fidelity and faithful to input images. We also propose a novel technique
to calculate more stable image-level gradients via diffusion models compared to
existing alternatives. Finally, we produce realistic animations by fine-tuning
the rendered shape and texture under rigid part transformations. Extensive
evaluations on multiple existing datasets as well as newly introduced noisy web
image collections with occlusions and truncation demonstrate that ARTIC3D
outputs are more robust to noisy images, higher quality in terms of shape and
texture details, and more realistic when animated. Project page:
https://chhankyao.github.io/artic3d/
ARTIC3D reconstructs 3D articulated shapes from complex monocular images using a skeleton-based model, 2D diffusion priors, and diffusion-guided optimization, achieving higher fidelity and more realistic animations.
skeleton-based surface representation 2D diffusion priors Stable Diffusion diffusion-guided 3D optimization image-level gradients diffusion models rigid part transformations