MOSPA: Human Motion Generation Driven by Spatial Audio
Enabling virtual humans to dynamically and realistically respond to diverse
auditory stimuli remains a key challenge in character animation, demanding the
integration of perceptual modeling and motion synthesis. Despite its
significance, this task remains largely unexplored. Most previous works have
primarily focused on mapping modalities like speech, audio, and music to
generate human motion. As of yet, these models typically overlook the impact of
spatial features encoded in spatial audio signals on human motion. To bridge
this gap and enable high-quality modeling of human movements in response to
spatial audio, we introduce the first comprehensive Spatial Audio-Driven Human
Motion (SAM) dataset, which contains diverse and high-quality spatial audio and
motion data. For benchmarking, we develop a simple yet effective
diffusion-based generative framework for human MOtion generation driven by
SPatial Audio, termed MOSPA, which faithfully captures the relationship between
body motion and spatial audio through an effective fusion mechanism. Once
trained, MOSPA could generate diverse realistic human motions conditioned on
varying spatial audio inputs. We perform a thorough investigation of the
proposed dataset and conduct extensive experiments for benchmarking, where our
method achieves state-of-the-art performance on this task. Our model and
dataset will be open-sourced upon acceptance. Please refer to our supplementary
video for more details.
687872c3001546c83aa4f9cf
Shuyang Xu
admin_assigned
2025-07-17T09:22:22.253Z
683d94e5ba11bab2cc848aab
/avatars/4a1915ad48c78b3a71733b3282f2f93c.svg
Shuyang Xu
user
JimSYXu
687872c3001546c83aa4f9d0
Zhiyang Dou
claimed_verified
2025-07-17T08:22:59.825Z
645223fb01d7bd9555ea399a
https://cdn-avatars.huggingface.co/v1/production/uploads/645223fb01d7bd9555ea399a/fVR7XmGg6pMSRKx9sEdvT.png
Zhiyang Dou
user
frankzydou
687872c3001546c83aa4f9d1
Mingyi Shi
687872c3001546c83aa4f9d2
Liang Pan
687872c3001546c83aa4f9d3
Leo Ho
687872c3001546c83aa4f9d4
Jingbo Wang
687872c3001546c83aa4f9d5
Yuan Liu
687872c3001546c83aa4f9d6
Cheng Lin
687872c3001546c83aa4f9d7
Yuexin Ma
687872c3001546c83aa4f9d8
Wenping Wang
687872c3001546c83aa4f9d9
Taku Komura
MOSPA: Human Motion Generation Driven by Spatial Audio
Enabling virtual humans to dynamically and realistically respond to diverse
auditory stimuli remains a key challenge in character animation, demanding the
integration of perceptual modeling and motion synthesis. Despite its
significance, this task remains largely unexplored. Most previous works have
primarily focused on mapping modalities like speech, audio, and music to
generate human motion. As of yet, these models typically overlook the impact of
spatial features encoded in spatial audio signals on human motion. To bridge
this gap and enable high-quality modeling of human movements in response to
spatial audio, we introduce the first comprehensive Spatial Audio-Driven Human
Motion (SAM) dataset, which contains diverse and high-quality spatial audio and
motion data. For benchmarking, we develop a simple yet effective
diffusion-based generative framework for human MOtion generation driven by
SPatial Audio, termed MOSPA, which faithfully captures the relationship between
body motion and spatial audio through an effective fusion mechanism. Once
trained, MOSPA could generate diverse realistic human motions conditioned on
varying spatial audio inputs. We perform a thorough investigation of the
proposed dataset and conduct extensive experiments for benchmarking, where our
method achieves state-of-the-art performance on this task. Our model and
dataset will be open-sourced upon acceptance. Please refer to our supplementary
video for more details.
A diffusion-based generative framework, MOSPA, is introduced to model human motion in response to spatial audio, achieving state-of-the-art performance using the newly created SAM dataset.
diffusion-based generative framework spatial audio human motion SAM dataset MOSPA spatial features perceptual modeling motion synthesis