SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware
  Video Generation
Audio-driven video generation aims to synthesize realistic videos that align
with input audio recordings, akin to the human ability to visualize scenes from
auditory input. However, existing approaches predominantly focus on exploring
semantic information, such as the classes of sounding sources present in the
audio, limiting their ability to generate videos with accurate content and
spatial composition. In contrast, we humans can not only naturally identify the
semantic categories of sounding sources but also determine their deeply encoded
spatial attributes, including locations and movement directions. This useful
information can be elucidated by considering specific spatial indicators
derived from the inherent physical properties of sound, such as loudness or
frequency. As prior methods largely ignore this factor, we present SpA2V, the
first framework explicitly exploits these spatial auditory cues from audios to
generate videos with high semantic and spatial correspondence. SpA2V decomposes
the generation process into two stages: 1) Audio-guided Video Planning: We
meticulously adapt a state-of-the-art MLLM for a novel task of harnessing
spatial and semantic cues from input audio to construct Video Scene Layouts
(VSLs). This serves as an intermediate representation to bridge the gap between
the audio and video modalities. 2) Layout-grounded Video Generation: We develop
an efficient and effective approach to seamlessly integrate VSLs as conditional
guidance into pre-trained diffusion models, enabling VSL-grounded video
generation in a training-free manner. Extensive experiments demonstrate that
SpA2V excels in generating realistic videos with semantic and spatial alignment
to the input audios.
689073e2e8049ec50047e173
Kien T. Pham
689073e2e8049ec50047e174
Yingqing He
689073e2e8049ec50047e175
Yazhou Xing
689073e2e8049ec50047e176
Qifeng Chen
689073e2e8049ec50047e177
Long Chen
SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware
  Video Generation
Audio-driven video generation aims to synthesize realistic videos that align
with input audio recordings, akin to the human ability to visualize scenes from
auditory input. However, existing approaches predominantly focus on exploring
semantic information, such as the classes of sounding sources present in the
audio, limiting their ability to generate videos with accurate content and
spatial composition. In contrast, we humans can not only naturally identify the
semantic categories of sounding sources but also determine their deeply encoded
spatial attributes, including locations and movement directions. This useful
information can be elucidated by considering specific spatial indicators
derived from the inherent physical properties of sound, such as loudness or
frequency. As prior methods largely ignore this factor, we present SpA2V, the
first framework explicitly exploits these spatial auditory cues from audios to
generate videos with high semantic and spatial correspondence. SpA2V decomposes
the generation process into two stages: 1) Audio-guided Video Planning: We
meticulously adapt a state-of-the-art MLLM for a novel task of harnessing
spatial and semantic cues from input audio to construct Video Scene Layouts
(VSLs). This serves as an intermediate representation to bridge the gap between
the audio and video modalities. 2) Layout-grounded Video Generation: We develop
an efficient and effective approach to seamlessly integrate VSLs as conditional
guidance into pre-trained diffusion models, enabling VSL-grounded video
generation in a training-free manner. Extensive experiments demonstrate that
SpA2V excels in generating realistic videos with semantic and spatial alignment
to the input audios.
SpA2V generates realistic videos aligned with input audio by leveraging spatial auditory cues and integrating them into diffusion models through video scene layouts.
MLLM Video Scene Layouts (VSLs) diffusion models audio-guided video planning layout-grounded video generation