title: MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware
    Diffusion and Iterative Refinement
summary: Existing works in single-image human reconstruction suffer from weak
  generalizability due to insufficient training data or 3D inconsistencies for a
  lack of comprehensive multi-view knowledge. In this paper, we introduce
  MagicMan, a human-specific multi-view diffusion model designed to generate
  high-quality novel view images from a single reference image. As its core, we
  leverage a pre-trained 2D diffusion model as the generative prior for
  generalizability, with the parametric SMPL-X model as the 3D body prior to
  promote 3D awareness. To tackle the critical challenge of maintaining
  consistency while achieving dense multi-view generation for improved 3D human
  reconstruction, we first introduce hybrid multi-view attention to facilitate
  both efficient and thorough information interchange across different views.
  Additionally, we present a geometry-aware dual branch to perform concurrent
  generation in both RGB and normal domains, further enhancing consistency via
  geometry cues. Last but not least, to address ill-shaped issues arising from
  inaccurate SMPL-X estimation that conflicts with the reference image, we
  propose a novel iterative refinement strategy, which progressively optimizes
  SMPL-X accuracy while enhancing the quality and consistency of the generated
  multi-views. Extensive experimental results demonstrate that our method
  significantly outperforms existing approaches in both novel view synthesis and
  subsequent 3D human reconstruction tasks.
paper_authors._id: 66cd4d50a7af45d8de3feb49
paper_authors.name: Xu He
paper_authors._id: 66cd4d50a7af45d8de3feb4a
paper_authors.name: Xiaoyu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T09:30:51.048Z
paper_authors._id: 66d8284b6bddfb32e77ddafb
paper_authors.avatarUrl: /avatars/edd371ecef6e7d58b945a30bbc6095ee.svg
paper_authors.fullname: Xiaoyu Li
paper_authors.type: user
paper_authors.user: Xiaoyu521
paper_authors._id: 66cd4d50a7af45d8de3feb4b
paper_authors.name: Di Kang
paper_authors._id: 66cd4d50a7af45d8de3feb4c
paper_authors.name: Jiangnan Ye
paper_authors._id: 66cd4d50a7af45d8de3feb4d
paper_authors.name: Chaopeng Zhang
paper_authors._id: 66cd4d50a7af45d8de3feb4e
paper_authors.name: Liyang Chen
paper_authors._id: 66cd4d50a7af45d8de3feb4f
paper_authors.name: Xiangjun Gao
paper_authors._id: 66cd4d50a7af45d8de3feb50
paper_authors.name: Han Zhang
paper_authors._id: 66cd4d50a7af45d8de3feb51
paper_authors.name: Zhiyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:16:00.427Z
paper_authors._id: 6280e830e99dccaac4bbfde5
paper_authors.avatarUrl: /avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
paper_authors.fullname: Zhiyong Wu
paper_authors.type: user
paper_authors.user: zy001
paper_authors._id: 66cd4d50a7af45d8de3feb52
paper_authors.name: Haolin Zhuang
paper_title: MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware
    Diffusion and Iterative Refinement
paper_summary: Existing works in single-image human reconstruction suffer from weak
  generalizability due to insufficient training data or 3D inconsistencies for a
  lack of comprehensive multi-view knowledge. In this paper, we introduce
  MagicMan, a human-specific multi-view diffusion model designed to generate
  high-quality novel view images from a single reference image. As its core, we
  leverage a pre-trained 2D diffusion model as the generative prior for
  generalizability, with the parametric SMPL-X model as the 3D body prior to
  promote 3D awareness. To tackle the critical challenge of maintaining
  consistency while achieving dense multi-view generation for improved 3D human
  reconstruction, we first introduce hybrid multi-view attention to facilitate
  both efficient and thorough information interchange across different views.
  Additionally, we present a geometry-aware dual branch to perform concurrent
  generation in both RGB and normal domains, further enhancing consistency via
  geometry cues. Last but not least, to address ill-shaped issues arising from
  inaccurate SMPL-X estimation that conflicts with the reference image, we
  propose a novel iterative refinement strategy, which progressively optimizes
  SMPL-X accuracy while enhancing the quality and consistency of the generated
  multi-views. Extensive experimental results demonstrate that our method
  significantly outperforms existing approaches in both novel view synthesis and
  subsequent 3D human reconstruction tasks.
paper_ai_keywords: diffusion model single-image human reconstruction single reference image novel view images pre-trained 2D diffusion model generative prior parametric SMPL-X model 3D body prior hybrid multi-view attention RGB domain normal domain geometry cues iterative refinement strategy SMPL-X estimation novel view synthesis 3D human reconstruction