title: UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified
    Model
summary: Audio-driven 3D facial animation aims to map input audio to realistic facial
  motion. Despite significant progress, limitations arise from inconsistent 3D
  annotations, restricting previous models to training on specific annotations
  and thereby constraining the training scale. In this work, we present
  UniTalker, a unified model featuring a multi-head architecture designed to
  effectively leverage datasets with varied annotations. To enhance training
  stability and ensure consistency among multi-head outputs, we employ three
  training strategies, namely, PCA, model warm-up, and pivot identity embedding.
  To expand the training scale and diversity, we assemble A2F-Bench, comprising
  five publicly available datasets and three newly curated datasets. These
  datasets contain a wide range of audio domains, covering multilingual speech
  voices and songs, thereby scaling the training data from commonly employed
  datasets, typically less than 1 hour, to 18.5 hours. With a single trained
  UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for
  BIWI dataset and 13.7% for Vocaset. Additionally, the pre-trained UniTalker
  exhibits promise as the foundation model for audio-driven facial animation
  tasks. Fine-tuning the pre-trained UniTalker on seen datasets further enhances
  performance on each dataset, with an average error reduction of 6.3% on
  A2F-Bench. Moreover, fine-tuning UniTalker on an unseen dataset with only half
  the data surpasses prior state-of-the-art models trained on the full dataset.
  The code and dataset are available at the project page
  https://github.com/X-niper/UniTalker.
paper_authors._id: 66ac4524c5ad63f8059976c5
paper_authors.name: Xiangyu Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T09:02:54.945Z
paper_authors._id: 657fba44bc9bceccf935ebfc
paper_authors.avatarUrl: /avatars/4bd7f2fc82963034fb1e86b8326cee28.svg
paper_authors.fullname: Xiangyu Fan
paper_authors.type: user
paper_authors.user: X-niper
paper_authors._id: 66ac4524c5ad63f8059976c6
paper_authors.name: Jiaqi Li
paper_authors._id: 66ac4524c5ad63f8059976c7
paper_authors.name: Zhiqian Lin
paper_authors._id: 66ac4524c5ad63f8059976c8
paper_authors.name: Weiye Xiao
paper_authors._id: 66ac4524c5ad63f8059976c9
paper_authors.name: Lei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T08:09:16.271Z
paper_authors._id: 6626a471430a124253f197c8
paper_authors.avatarUrl: /avatars/f5747fdbe495d1296fed9d16d8c95857.svg
paper_authors.fullname: yl-1993
paper_authors.type: user
paper_authors.user: yl-1993
paper_title: UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified
    Model
paper_summary: Audio-driven 3D facial animation aims to map input audio to realistic facial
  motion. Despite significant progress, limitations arise from inconsistent 3D
  annotations, restricting previous models to training on specific annotations
  and thereby constraining the training scale. In this work, we present
  UniTalker, a unified model featuring a multi-head architecture designed to
  effectively leverage datasets with varied annotations. To enhance training
  stability and ensure consistency among multi-head outputs, we employ three
  training strategies, namely, PCA, model warm-up, and pivot identity embedding.
  To expand the training scale and diversity, we assemble A2F-Bench, comprising
  five publicly available datasets and three newly curated datasets. These
  datasets contain a wide range of audio domains, covering multilingual speech
  voices and songs, thereby scaling the training data from commonly employed
  datasets, typically less than 1 hour, to 18.5 hours. With a single trained
  UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for
  BIWI dataset and 13.7% for Vocaset. Additionally, the pre-trained UniTalker
  exhibits promise as the foundation model for audio-driven facial animation
  tasks. Fine-tuning the pre-trained UniTalker on seen datasets further enhances
  performance on each dataset, with an average error reduction of 6.3% on
  A2F-Bench. Moreover, fine-tuning UniTalker on an unseen dataset with only half
  the data surpasses prior state-of-the-art models trained on the full dataset.
  The code and dataset are available at the project page
  https://github.com/X-niper/UniTalker.
paper_ai_keywords: multi-head architecture PCA model warm-up pivot identity embedding A2F-Bench lip vertex error BIWI dataset Vocaset fine-tuning