Multi-human Interactive Talking Dataset
Existing studies on talking video generation have predominantly focused on
single-person monologues or isolated facial animations, limiting their
applicability to realistic multi-human interactions. To bridge this gap, we
introduce MIT, a large-scale dataset specifically designed for multi-human
talking video generation. To this end, we develop an automatic pipeline that
collects and annotates multi-person conversational videos. The resulting
dataset comprises 12 hours of high-resolution footage, each featuring two to
four speakers, with fine-grained annotations of body poses and speech
interactions. It captures natural conversational dynamics in multi-speaker
scenario, offering a rich resource for studying interactive visual behaviors.
To demonstrate the potential of MIT, we furthur propose CovOG, a baseline model
for this novel task. It integrates a Multi-Human Pose Encoder (MPE) to handle
varying numbers of speakers by aggregating individual pose embeddings, and an
Interactive Audio Driver (IAD) to modulate head dynamics based on
speaker-specific audio features. Together, these components showcase the
feasibility and challenges of generating realistic multi-human talking videos,
establishing MIT as a valuable benchmark for future research. The code is
avalibale at: https://github.com/showlab/Multi-human-Talking-Video-Dataset.
6892e5018da45ffb0a2b24dc
Zeyu Zhu
6892e5018da45ffb0a2b24dd
Weijia Wu
6892e5018da45ffb0a2b24de
Mike Zheng Shou
Multi-human Interactive Talking Dataset
Existing studies on talking video generation have predominantly focused on
single-person monologues or isolated facial animations, limiting their
applicability to realistic multi-human interactions. To bridge this gap, we
introduce MIT, a large-scale dataset specifically designed for multi-human
talking video generation. To this end, we develop an automatic pipeline that
collects and annotates multi-person conversational videos. The resulting
dataset comprises 12 hours of high-resolution footage, each featuring two to
four speakers, with fine-grained annotations of body poses and speech
interactions. It captures natural conversational dynamics in multi-speaker
scenario, offering a rich resource for studying interactive visual behaviors.
To demonstrate the potential of MIT, we furthur propose CovOG, a baseline model
for this novel task. It integrates a Multi-Human Pose Encoder (MPE) to handle
varying numbers of speakers by aggregating individual pose embeddings, and an
Interactive Audio Driver (IAD) to modulate head dynamics based on
speaker-specific audio features. Together, these components showcase the
feasibility and challenges of generating realistic multi-human talking videos,
establishing MIT as a valuable benchmark for future research. The code is
avalibale at: https://github.com/showlab/Multi-human-Talking-Video-Dataset.
MIT, a large-scale dataset for multi-human talking video generation, includes fine-grained annotations and is used to demonstrate CovOG, a baseline model integrating a Multi-Human Pose Encoder and an Interactive Audio Driver.
Multi-Human Pose Encoder Interactive Audio Driver multi-human talking video generation fine-grained annotations body poses speech interactions conversational dynamics multi-speaker scenario