MusicInfuser: Making Video Diffusion Listen and Dance
We introduce MusicInfuser, an approach for generating high-quality dance
videos that are synchronized to a specified music track. Rather than attempting
to design and train a new multimodal audio-video model, we show how existing
video diffusion models can be adapted to align with musical inputs by
introducing lightweight music-video cross-attention and a low-rank adapter.
Unlike prior work requiring motion capture data, our approach fine-tunes only
on dance videos. MusicInfuser achieves high-quality music-driven video
generation while preserving the flexibility and generative capabilities of the
underlying models. We introduce an evaluation framework using Video-LLMs to
assess multiple dimensions of dance generation quality. The project page and
code are available at https://susunghong.github.io/MusicInfuser.
67db13f71956dcedf0b4d357
Susung Hong
admin_assigned
2025-03-20T13:25:56.403Z
635a6dd21668c4ead3ed19fa
https://cdn-avatars.huggingface.co/v1/production/uploads/1679159537671-635a6dd21668c4ead3ed19fa.jpeg
Susung Hong
user
susunghong
67db13f71956dcedf0b4d358
Ira Kemelmacher-Shlizerman
admin_assigned
2025-03-20T13:26:06.351Z
66d20e411ba71ac4c0488132
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kkqDAFT15lx4wrJVXiFGJ.png
Ira Kemelmacher-Shlizerman
user
kemelmi
67db13f71956dcedf0b4d359
Brian Curless
67db13f71956dcedf0b4d35a
Steven M. Seitz
MusicInfuser: Making Video Diffusion Listen and Dance
We introduce MusicInfuser, an approach for generating high-quality dance
videos that are synchronized to a specified music track. Rather than attempting
to design and train a new multimodal audio-video model, we show how existing
video diffusion models can be adapted to align with musical inputs by
introducing lightweight music-video cross-attention and a low-rank adapter.
Unlike prior work requiring motion capture data, our approach fine-tunes only
on dance videos. MusicInfuser achieves high-quality music-driven video
generation while preserving the flexibility and generative capabilities of the
underlying models. We introduce an evaluation framework using Video-LLMs to
assess multiple dimensions of dance generation quality. The project page and
code are available at https://susunghong.github.io/MusicInfuser.
MusicInfuser uses adapted video diffusion models with music-video cross-attention and low-rank adapters to generate high-quality dance videos synchronized to music without motion capture data.
video diffusion models music-video cross-attention low-rank adapter Video-LLMs