SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual
  Dyadic Interactive Human Generation
The rapid development of large-scale models has catalyzed significant
breakthroughs in the digital human domain. These advanced methodologies offer
high-fidelity solutions for avatar driving and rendering, leading academia to
focus on the next major challenge: audio-visual dyadic interactive virtual
human. To facilitate research in this emerging area, we present SpeakerVid-5M
dataset, the first large-scale, high-quality dataset designed for audio-visual
dyadic interactive virtual human generation. Totaling over 8,743 hours,
SpeakerVid-5M contains more than 5.2 million video clips of human portraits. It
covers diverse scales and interaction types, including monadic talking,
listening, and dyadic conversations. Crucially, the dataset is structured along
two key dimensions: interaction type and data quality. First, it is categorized
into four types (dialogue branch, single branch, listening branch and
multi-turn branch) based on the interaction scenario. Second, it is stratified
into a large-scale pre-training subset and a curated, high-quality subset for
Supervised Fine-Tuning (SFT). This dual structure accommodates a wide array of
2D virtual human tasks. In addition, we provide an autoregressive (AR)-based
video chat baseline trained on this data, accompanied by a dedicated set of
metrics and test data to serve as a benchmark VidChatBench for future work.
Both the dataset and the corresponding data processing code will be publicly
released. Project page: https://dorniwang.github.io/SpeakerVid-5M/
6875c14a257d4f043537056b
Youliang Zhang
6875c14a257d4f043537056c
Zhaoyang Li
6875c14a257d4f043537056d
Duomin Wang
claimed_verified
2025-07-15T19:09:37.361Z
64ae9b88a22a179fc4d07992
/avatars/c9065f04a1188ea3129e56a90328ffd3.svg
wang
user
dorni
6875c14a257d4f043537056e
Jiahe Zhang
6875c14a257d4f043537056f
Deyu Zhou
6875c14a257d4f0435370570
Zixin Yin
6875c14a257d4f0435370571
Xili Dai
6875c14a257d4f0435370572
Gang Yu
6875c14a257d4f0435370573
Xiu Li
SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual
  Dyadic Interactive Human Generation
The rapid development of large-scale models has catalyzed significant
breakthroughs in the digital human domain. These advanced methodologies offer
high-fidelity solutions for avatar driving and rendering, leading academia to
focus on the next major challenge: audio-visual dyadic interactive virtual
human. To facilitate research in this emerging area, we present SpeakerVid-5M
dataset, the first large-scale, high-quality dataset designed for audio-visual
dyadic interactive virtual human generation. Totaling over 8,743 hours,
SpeakerVid-5M contains more than 5.2 million video clips of human portraits. It
covers diverse scales and interaction types, including monadic talking,
listening, and dyadic conversations. Crucially, the dataset is structured along
two key dimensions: interaction type and data quality. First, it is categorized
into four types (dialogue branch, single branch, listening branch and
multi-turn branch) based on the interaction scenario. Second, it is stratified
into a large-scale pre-training subset and a curated, high-quality subset for
Supervised Fine-Tuning (SFT). This dual structure accommodates a wide array of
2D virtual human tasks. In addition, we provide an autoregressive (AR)-based
video chat baseline trained on this data, accompanied by a dedicated set of
metrics and test data to serve as a benchmark VidChatBench for future work.
Both the dataset and the corresponding data processing code will be publicly
released. Project page: https://dorniwang.github.io/SpeakerVid-5M/
SpeakerVid-5M is a large-scale, high-quality dataset for audio-visual dyadic interactive virtual human generation, featuring diverse interaction types and data quality levels, along with an autoregressive video chat baseline and benchmark metrics.
audio-visual dyadic interactive virtual human SpeakerVid-5M video clips interaction type dialogue branch single branch listening branch multi-turn branch pre-training subset Supervised Fine-Tuning autoregressive video chat baseline VidChatBench