NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interaction
<|reference_start|>NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of Hand-Object Interaction: Modeling hand-object interactions is a fundamentally challenging task in 3D computer vision. Despite remarkable progress that has been achieved in this field, existing methods still fail to synthesize the hand-object interaction photo-realistically, suffering from degraded rendering quality caused by the heavy mutual occlusions between the hand and the object, and inaccurate hand-object pose estimation. To tackle these challenges, we present a novel free-viewpoint rendering framework, Neural Contact Radiance Field (NCRF), to reconstruct hand-object interactions from a sparse set of videos. In particular, the proposed NCRF framework consists of two key components: (a) A contact optimization field that predicts an accurate contact field from 3D query points for achieving desirable contact between the hand and the object. (b) A hand-object neural radiance field to learn an implicit hand-object representation in a static canonical space, in concert with the specifically designed hand-object motion field to produce observation-to-canonical correspondences. We jointly learn these key components where they mutually help and regularize each other with visual and geometric constraints, producing a high-quality hand-object reconstruction that achieves photo-realistic novel view synthesis. Extensive experiments on HO3D and DexYCB datasets show that our approach outperforms the current state-of-the-art in terms of both rendering quality and pose estimation accuracy.<|reference_end|>
@article{zhang2024ncrf:,
    title={NCRF: Neural Contact Radiance Fields for Free-Viewpoint Rendering of
  Hand-Object Interaction},
    author={Zhongqun Zhang and Jifei Song and Eduardo P'erez-Pellitero and Yiren
  Zhou and Hyung Jin Chang and Alev{s} Leonardis},
    journal={arXiv preprint arXiv:2402.05532},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.05532},
    primaryClass={cs.CV}
}