  This paper presents FaceXHuBERT, a text-less speech-driven 3D facial
animation generation method that allows to capture personalized and subtle cues
in speech (e.g. identity, emotion and hesitation). It is also very robust to
background noise and can handle audio recorded in a variety of situations (e.g.
multiple people speaking). Recent approaches employ end-to-end deep learning
taking into account both audio and text as input to generate facial animation
for the whole face. However, scarcity of publicly available expressive audio-3D
facial animation datasets poses a major bottleneck. The resulting animations
still have issues regarding accurate lip-synching, expressivity,
person-specific information and generalizability. We effectively employ
self-supervised pretrained HuBERT model in the training process that allows us
to incorporate both lexical and non-lexical information in the audio without
using a large lexicon. Additionally, guiding the training with a binary emotion
condition and speaker identity distinguishes the tiniest subtle facial motion.
We carried out extensive objective and subjective evaluation in comparison to
ground-truth and state-of-the-art work. A perceptual user study demonstrates
that our approach produces superior results with respect to the realism of the
animation 78% of the time in comparison to the state-of-the-art. In addition,
our method is 4 times faster eliminating the use of complex sequential models
such as transformers. We strongly recommend watching the supplementary video
before reading the paper. We also provide the implementation and evaluation
codes with a GitHub repository link.

Computer Vision and Pattern Recognition;Artificial Intelligence