title: MusicInfuser: Making Video Diffusion Listen and Dance
summary: We introduce MusicInfuser, an approach for generating high-quality dance
  videos that are synchronized to a specified music track. Rather than attempting
  to design and train a new multimodal audio-video model, we show how existing
  video diffusion models can be adapted to align with musical inputs by
  introducing lightweight music-video cross-attention and a low-rank adapter.
  Unlike prior work requiring motion capture data, our approach fine-tunes only
  on dance videos. MusicInfuser achieves high-quality music-driven video
  generation while preserving the flexibility and generative capabilities of the
  underlying models. We introduce an evaluation framework using Video-LLMs to
  assess multiple dimensions of dance generation quality. The project page and
  code are available at https://susunghong.github.io/MusicInfuser.
paper_authors._id: 67db13f71956dcedf0b4d357
paper_authors.name: Susung Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:25:56.403Z
paper_authors._id: 635a6dd21668c4ead3ed19fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679159537671-635a6dd21668c4ead3ed19fa.jpeg
paper_authors.fullname: Susung Hong
paper_authors.type: user
paper_authors.user: susunghong
paper_authors._id: 67db13f71956dcedf0b4d358
paper_authors.name: Ira Kemelmacher-Shlizerman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:26:06.351Z
paper_authors._id: 66d20e411ba71ac4c0488132
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kkqDAFT15lx4wrJVXiFGJ.png
paper_authors.fullname: Ira Kemelmacher-Shlizerman
paper_authors.type: user
paper_authors.user: kemelmi
paper_authors._id: 67db13f71956dcedf0b4d359
paper_authors.name: Brian Curless
paper_authors._id: 67db13f71956dcedf0b4d35a
paper_authors.name: Steven M. Seitz
paper_title: MusicInfuser: Making Video Diffusion Listen and Dance
paper_summary: We introduce MusicInfuser, an approach for generating high-quality dance
  videos that are synchronized to a specified music track. Rather than attempting
  to design and train a new multimodal audio-video model, we show how existing
  video diffusion models can be adapted to align with musical inputs by
  introducing lightweight music-video cross-attention and a low-rank adapter.
  Unlike prior work requiring motion capture data, our approach fine-tunes only
  on dance videos. MusicInfuser achieves high-quality music-driven video
  generation while preserving the flexibility and generative capabilities of the
  underlying models. We introduce an evaluation framework using Video-LLMs to
  assess multiple dimensions of dance generation quality. The project page and
  code are available at https://susunghong.github.io/MusicInfuser.
paper_ai_summary: MusicInfuser uses adapted video diffusion models with music-video cross-attention and low-rank adapters to generate high-quality dance videos synchronized to music without motion capture data.
paper_ai_keywords: video diffusion models music-video cross-attention low-rank adapter Video-LLMs