UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified
  Model
Audio-driven 3D facial animation aims to map input audio to realistic facial
motion. Despite significant progress, limitations arise from inconsistent 3D
annotations, restricting previous models to training on specific annotations
and thereby constraining the training scale. In this work, we present
UniTalker, a unified model featuring a multi-head architecture designed to
effectively leverage datasets with varied annotations. To enhance training
stability and ensure consistency among multi-head outputs, we employ three
training strategies, namely, PCA, model warm-up, and pivot identity embedding.
To expand the training scale and diversity, we assemble A2F-Bench, comprising
five publicly available datasets and three newly curated datasets. These
datasets contain a wide range of audio domains, covering multilingual speech
voices and songs, thereby scaling the training data from commonly employed
datasets, typically less than 1 hour, to 18.5 hours. With a single trained
UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for
BIWI dataset and 13.7% for Vocaset. Additionally, the pre-trained UniTalker
exhibits promise as the foundation model for audio-driven facial animation
tasks. Fine-tuning the pre-trained UniTalker on seen datasets further enhances
performance on each dataset, with an average error reduction of 6.3% on
A2F-Bench. Moreover, fine-tuning UniTalker on an unseen dataset with only half
the data surpasses prior state-of-the-art models trained on the full dataset.
The code and dataset are available at the project page
https://github.com/X-niper/UniTalker.
66ac4524c5ad63f8059976c5
Xiangyu Fan
admin_assigned
2024-08-02T09:02:54.945Z
657fba44bc9bceccf935ebfc
/avatars/4bd7f2fc82963034fb1e86b8326cee28.svg
Xiangyu Fan
user
X-niper
66ac4524c5ad63f8059976c6
Jiaqi Li
66ac4524c5ad63f8059976c7
Zhiqian Lin
66ac4524c5ad63f8059976c8
Weiye Xiao
66ac4524c5ad63f8059976c9
Lei Yang
admin_assigned
2025-03-10T08:09:16.271Z
6626a471430a124253f197c8
/avatars/f5747fdbe495d1296fed9d16d8c95857.svg
yl-1993
user
yl-1993
UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified
  Model
Audio-driven 3D facial animation aims to map input audio to realistic facial
motion. Despite significant progress, limitations arise from inconsistent 3D
annotations, restricting previous models to training on specific annotations
and thereby constraining the training scale. In this work, we present
UniTalker, a unified model featuring a multi-head architecture designed to
effectively leverage datasets with varied annotations. To enhance training
stability and ensure consistency among multi-head outputs, we employ three
training strategies, namely, PCA, model warm-up, and pivot identity embedding.
To expand the training scale and diversity, we assemble A2F-Bench, comprising
five publicly available datasets and three newly curated datasets. These
datasets contain a wide range of audio domains, covering multilingual speech
voices and songs, thereby scaling the training data from commonly employed
datasets, typically less than 1 hour, to 18.5 hours. With a single trained
UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for
BIWI dataset and 13.7% for Vocaset. Additionally, the pre-trained UniTalker
exhibits promise as the foundation model for audio-driven facial animation
tasks. Fine-tuning the pre-trained UniTalker on seen datasets further enhances
performance on each dataset, with an average error reduction of 6.3% on
A2F-Bench. Moreover, fine-tuning UniTalker on an unseen dataset with only half
the data surpasses prior state-of-the-art models trained on the full dataset.
The code and dataset are available at the project page
https://github.com/X-niper/UniTalker.
UniTalker, a unified multi-head model, improves audio-driven 3D facial animation through varied dataset training and optimization strategies, achieving superior performance across multiple datasets.
multi-head architecture PCA model warm-up pivot identity embedding A2F-Bench lips vertex error BIWI dataset Vocaset pre-trained fine-tuning