MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware
  Diffusion and Iterative Refinement
Existing works in single-image human reconstruction suffer from weak
generalizability due to insufficient training data or 3D inconsistencies for a
lack of comprehensive multi-view knowledge. In this paper, we introduce
MagicMan, a human-specific multi-view diffusion model designed to generate
high-quality novel view images from a single reference image. As its core, we
leverage a pre-trained 2D diffusion model as the generative prior for
generalizability, with the parametric SMPL-X model as the 3D body prior to
promote 3D awareness. To tackle the critical challenge of maintaining
consistency while achieving dense multi-view generation for improved 3D human
reconstruction, we first introduce hybrid multi-view attention to facilitate
both efficient and thorough information interchange across different views.
Additionally, we present a geometry-aware dual branch to perform concurrent
generation in both RGB and normal domains, further enhancing consistency via
geometry cues. Last but not least, to address ill-shaped issues arising from
inaccurate SMPL-X estimation that conflicts with the reference image, we
propose a novel iterative refinement strategy, which progressively optimizes
SMPL-X accuracy while enhancing the quality and consistency of the generated
multi-views. Extensive experimental results demonstrate that our method
significantly outperforms existing approaches in both novel view synthesis and
subsequent 3D human reconstruction tasks.
66cd4d50a7af45d8de3feb49
Xu He
66cd4d50a7af45d8de3feb4a
Xiaoyu Li
claimed_verified
2024-09-04T09:30:51.048Z
66d8284b6bddfb32e77ddafb
/avatars/edd371ecef6e7d58b945a30bbc6095ee.svg
Xiaoyu Li
user
Xiaoyu521
66cd4d50a7af45d8de3feb4b
Di Kang
66cd4d50a7af45d8de3feb4c
Jiangnan Ye
66cd4d50a7af45d8de3feb4d
Chaopeng Zhang
66cd4d50a7af45d8de3feb4e
Liyang Chen
66cd4d50a7af45d8de3feb4f
Xiangjun Gao
66cd4d50a7af45d8de3feb50
Han Zhang
66cd4d50a7af45d8de3feb51
Zhiyong Wu
admin_assigned
2024-08-27T13:16:00.427Z
6280e830e99dccaac4bbfde5
/avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
Zhiyong Wu
user
zy001
66cd4d50a7af45d8de3feb52
Haolin Zhuang
MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware
  Diffusion and Iterative Refinement
Existing works in single-image human reconstruction suffer from weak
generalizability due to insufficient training data or 3D inconsistencies for a
lack of comprehensive multi-view knowledge. In this paper, we introduce
MagicMan, a human-specific multi-view diffusion model designed to generate
high-quality novel view images from a single reference image. As its core, we
leverage a pre-trained 2D diffusion model as the generative prior for
generalizability, with the parametric SMPL-X model as the 3D body prior to
promote 3D awareness. To tackle the critical challenge of maintaining
consistency while achieving dense multi-view generation for improved 3D human
reconstruction, we first introduce hybrid multi-view attention to facilitate
both efficient and thorough information interchange across different views.
Additionally, we present a geometry-aware dual branch to perform concurrent
generation in both RGB and normal domains, further enhancing consistency via
geometry cues. Last but not least, to address ill-shaped issues arising from
inaccurate SMPL-X estimation that conflicts with the reference image, we
propose a novel iterative refinement strategy, which progressively optimizes
SMPL-X accuracy while enhancing the quality and consistency of the generated
multi-views. Extensive experimental results demonstrate that our method
significantly outperforms existing approaches in both novel view synthesis and
subsequent 3D human reconstruction tasks.
MagicMan, a multi-view diffusion model using SMPL-X and hybrid multi-view attention, generates consistent high-quality novel views and enhances 3D reconstruction accuracy with an iterative refinement strategy.
multi-view diffusion model SMPL-X model hybrid multi-view attention geometry-aware dual branch iterative refinement strategy novel view synthesis 3D human reconstruction