title: Single-Image 3D Human Digitization with Shape-Guided Diffusion
summary: We present an approach to generate a 360-degree view of a person with a
  consistent, high-resolution appearance from a single input image. NeRF and its
  variants typically require videos or images from different viewpoints. Most
  existing approaches taking monocular input either rely on ground-truth 3D scans
  for supervision or lack 3D consistency. While recent 3D generative models show
  promise of 3D consistent human digitization, these approaches do not generalize
  well to diverse clothing appearances, and the results lack photorealism. Unlike
  existing work, we utilize high-capacity 2D diffusion models pretrained for
  general image synthesis tasks as an appearance prior of clothed humans. To
  achieve better 3D consistency while retaining the input identity, we
  progressively synthesize multiple views of the human in the input image by
  inpainting missing regions with shape-guided diffusion conditioned on
  silhouette and surface normal. We then fuse these synthesized multi-view images
  via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
  given person. Experiments show that our approach outperforms prior methods and
  achieves photorealistic 360-degree synthesis of a wide range of clothed humans
  with complex textures from a single image.
paper_authors._id: 6555b04e4f361968f0fcebc0
paper_authors.name: Badour AlBahar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-16T15:51:07.035Z
paper_authors._id: 635d39efd6bbbf54119a76a0
paper_authors.avatarUrl: /avatars/97e2dca56fe72ed7406acc52180be409.svg
paper_authors.fullname: Badour Albahar
paper_authors.type: user
paper_authors.user: Badour
paper_authors._id: 6555b04e4f361968f0fcebc1
paper_authors.name: Shunsuke Saito
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:12:40.028Z
paper_authors._id: 630688edd37ce67e0e50e548
paper_authors.avatarUrl: /avatars/1681e465d7649f67f94e1b69d236cb1e.svg
paper_authors.fullname: Shunsuke Saito
paper_authors.type: user
paper_authors.user: psyth
paper_authors._id: 6555b04e4f361968f0fcebc2
paper_authors.name: Hung-Yu Tseng
paper_authors._id: 6555b04e4f361968f0fcebc3
paper_authors.name: Changil Kim
paper_authors._id: 6555b04e4f361968f0fcebc4
paper_authors.name: Johannes Kopf
paper_authors._id: 6555b04e4f361968f0fcebc5
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:14:16.285Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_title: Single-Image 3D Human Digitization with Shape-Guided Diffusion
paper_summary: We present an approach to generate a 360-degree view of a person with a
  consistent, high-resolution appearance from a single input image. NeRF and its
  variants typically require videos or images from different viewpoints. Most
  existing approaches taking monocular input either rely on ground-truth 3D scans
  for supervision or lack 3D consistency. While recent 3D generative models show
  promise of 3D consistent human digitization, these approaches do not generalize
  well to diverse clothing appearances, and the results lack photorealism. Unlike
  existing work, we utilize high-capacity 2D diffusion models pretrained for
  general image synthesis tasks as an appearance prior of clothed humans. To
  achieve better 3D consistency while retaining the input identity, we
  progressively synthesize multiple views of the human in the input image by
  inpainting missing regions with shape-guided diffusion conditioned on
  silhouette and surface normal. We then fuse these synthesized multi-view images
  via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
  given person. Experiments show that our approach outperforms prior methods and
  achieves photorealistic 360-degree synthesis of a wide range of clothed humans
  with complex textures from a single image.
paper_ai_summary: An approach using 2D diffusion models and inverse rendering generates photorealistic 360-degree views of people from single images.
paper_ai_keywords: NeRF 3D generative models 2D diffusion models appearance prior shape-guided diffusion silhouette surface normal inverse rendering 3D mesh