MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement
<|reference_start|>MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement: Existing works in single-image human reconstruction suffer from weak generalizability due to insufficient training data or 3D inconsistencies for a lack of comprehensive multi-view knowledge. In this paper, we introduce MagicMan, a human-specific multi-view diffusion model designed to generate high-quality novel view images from a single reference image. As its core, we leverage a pre-trained 2D diffusion model as the generative prior for generalizability, with the parametric SMPL-X model as the 3D body prior to promote 3D awareness. To tackle the critical challenge of maintaining consistency while achieving dense multi-view generation for improved 3D human reconstruction, we first introduce hybrid multi-view attention to facilitate both efficient and thorough information interchange across different views. Additionally, we present a geometry-aware dual branch to perform concurrent generation in both RGB and normal domains, further enhancing consistency via geometry cues. Last but not least, to address ill-shaped issues arising from inaccurate SMPL-X estimation that conflicts with the reference image, we propose a novel iterative refinement strategy, which progressively optimizes SMPL-X accuracy while enhancing the quality and consistency of the generated multi-views. Extensive experimental results demonstrate that our method significantly outperforms existing approaches in both novel view synthesis and subsequent 3D human reconstruction tasks.<|reference_end|>
@article{he2024magicman:,
    title={MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware
  Diffusion and Iterative Refinement},
    author={Xu He and Xiaoyu Li and Di Kang and Jiangnan Ye and Chaopeng Zhang and
  Liyang Chen and Xiangjun Gao and Han Zhang and Zhiyong Wu and Haolin Zhuang},
    journal={arXiv preprint arXiv:2408.14211},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.14211},
    primaryClass={cs.CV cs.AI}
}