Coherent 3D Portrait Video Reconstruction via Triplane Fusion
<|reference_start|>Coherent 3D Portrait Video Reconstruction via Triplane Fusion: Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, potentially democratizing telepresence. However, per-frame 3D reconstruction exhibits temporal inconsistency and forgets the user's appearance. On the other hand, self-reenactment methods can render coherent 3D portraits by driving a personalized 3D prior, but fail to faithfully reconstruct the user's per-frame appearance (e.g., facial expressions and lighting). In this work, we recognize the need to maintain both coherent identity and dynamic per-frame appearance to enable the best possible realism. To this end, we propose a new fusion-based method that fuses a personalized 3D subject prior with per-frame information, producing temporally stable 3D videos with faithful reconstruction of the user's per-frame appearances. Trained only using synthetic data produced by an expression-conditioned 3D GAN, our encoder-based method achieves both state-of-the-art 3D reconstruction accuracy and temporal consistency on in-studio and in-the-wild datasets.<|reference_end|>
@article{wang2024coherent,
    title={Coherent 3D Portrait Video Reconstruction via Triplane Fusion},
    author={Shengze Wang, Xueting Li, Chao Liu, Matthew Chan, Michael Stengel,
  Josef Spjut, Henry Fuchs, Shalini De Mello, Koki Nagano},
    journal={arXiv preprint arXiv:2405.00794},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00794},
    primaryClass={cs.CV}
}