  Generating high-fidelity talking head video by fitting with the input audio
sequence is a challenging problem that receives considerable attentions
recently. In this paper, we address this problem with the aid of neural scene
representation networks. Our method is completely different from existing
methods that rely on intermediate representations like 2D landmarks or 3D face
models to bridge the gap between audio input and video output. Specifically,
the feature of input audio signal is directly fed into a conditional implicit
function to generate a dynamic neural radiance field, from which a
high-fidelity talking-head video corresponding to the audio signal is
synthesized using volume rendering. Another advantage of our framework is that
not only the head (with hair) region is synthesized as previous methods did,
but also the upper body is generated via two individual neural radiance fields.
Experimental results demonstrate that our novel framework can (1) produce
high-fidelity and natural results, and (2) support free adjustment of audio
signals, viewing directions, and background images. Code is available at
https://github.com/YudongGuo/AD-NeRF.

Computer Vision and Pattern Recognition