title: ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image
    Collections
summary: Estimating 3D articulated shapes like animal bodies from monocular images is
  inherently challenging due to the ambiguities of camera viewpoint, pose,
  texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to
  reconstruct per-instance 3D shapes from a sparse image collection in-the-wild.
  Specifically, ARTIC3D is built upon a skeleton-based surface representation and
  is further guided by 2D diffusion priors from Stable Diffusion. First, we
  enhance the input images with occlusions/truncation via 2D diffusion to obtain
  cleaner mask estimates and semantic features. Second, we perform
  diffusion-guided 3D optimization to estimate shape and texture that are of
  high-fidelity and faithful to input images. We also propose a novel technique
  to calculate more stable image-level gradients via diffusion models compared to
  existing alternatives. Finally, we produce realistic animations by fine-tuning
  the rendered shape and texture under rigid part transformations. Extensive
  evaluations on multiple existing datasets as well as newly introduced noisy web
  image collections with occlusions and truncation demonstrate that ARTIC3D
  outputs are more robust to noisy images, higher quality in terms of shape and
  texture details, and more realistic when animated. Project page:
  https://chhankyao.github.io/artic3d/
paper_authors._id: 648145c5de559d48dbb551ec
paper_authors.name: Chun-Han Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:50:25.340Z
paper_authors._id: 63d9b9c9aefaa73525854853
paper_authors.avatarUrl: /avatars/166b59f9cdd0998b354d013528c01add.svg
paper_authors.fullname: Chun-Han Yao
paper_authors.type: user
paper_authors.user: cyao6
paper_authors._id: 648145c5de559d48dbb551ed
paper_authors.name: Amit Raj
paper_authors._id: 648145c5de559d48dbb551ee
paper_authors.name: Wei-Chih Hung
paper_authors._id: 648145c5de559d48dbb551ef
paper_authors.name: Yuanzhen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:50:59.654Z
paper_authors._id: 6334ee5831a2be3938c85639
paper_authors.avatarUrl: /avatars/9821af26a585f54b595ec39f061f2250.svg
paper_authors.fullname: Yuanzhen Li
paper_authors.type: user
paper_authors.user: yzli
paper_authors._id: 648145c5de559d48dbb551f0
paper_authors.name: Michael Rubinstein
paper_authors._id: 648145c5de559d48dbb551f1
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 648145c5de559d48dbb551f2
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:51:19.349Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_title: ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image
    Collections
paper_summary: Estimating 3D articulated shapes like animal bodies from monocular images is
  inherently challenging due to the ambiguities of camera viewpoint, pose,
  texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to
  reconstruct per-instance 3D shapes from a sparse image collection in-the-wild.
  Specifically, ARTIC3D is built upon a skeleton-based surface representation and
  is further guided by 2D diffusion priors from Stable Diffusion. First, we
  enhance the input images with occlusions/truncation via 2D diffusion to obtain
  cleaner mask estimates and semantic features. Second, we perform
  diffusion-guided 3D optimization to estimate shape and texture that are of
  high-fidelity and faithful to input images. We also propose a novel technique
  to calculate more stable image-level gradients via diffusion models compared to
  existing alternatives. Finally, we produce realistic animations by fine-tuning
  the rendered shape and texture under rigid part transformations. Extensive
  evaluations on multiple existing datasets as well as newly introduced noisy web
  image collections with occlusions and truncation demonstrate that ARTIC3D
  outputs are more robust to noisy images, higher quality in terms of shape and
  texture details, and more realistic when animated. Project page:
  https://chhankyao.github.io/artic3d/
paper_ai_summary: ARTIC3D reconstructs 3D articulated shapes from complex monocular images using a skeleton-based model, 2D diffusion priors, and diffusion-guided optimization, achieving higher fidelity and more realistic animations.
paper_ai_keywords: skeleton-based surface representation 2D diffusion priors Stable Diffusion diffusion-guided 3D optimization image-level gradients diffusion models rigid part transformations