title: MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic
    Synchronization
summary: Generating music that aligns with the visual content of a video has been a
  challenging task, as it requires a deep understanding of visual semantics and
  involves generating music whose melody, rhythm, and dynamics harmonize with the
  visual narratives. This paper presents MuVi, a novel framework that effectively
  addresses these challenges to enhance the cohesion and immersive experience of
  audio-visual content. MuVi analyzes video content through a specially designed
  visual adaptor to extract contextually and temporally relevant features. These
  features are used to generate music that not only matches the video's mood and
  theme but also its rhythm and pacing. We also introduce a contrastive
  music-visual pre-training scheme to ensure synchronization, based on the
  periodicity nature of music phrases. In addition, we demonstrate that our
  flow-matching-based music generator has in-context learning ability, allowing
  us to control the style and genre of the generated music. Experimental results
  show that MuVi demonstrates superior performance in both audio quality and
  temporal synchronization. The generated music video samples are available at
  https://muvi-v2m.github.io.
paper_authors._id: 6712157b48e62600e97219d8
paper_authors.name: Ruiqi Li
paper_authors._id: 6712157b48e62600e97219d9
paper_authors.name: Siqi Zheng
paper_authors._id: 6712157b48e62600e97219da
paper_authors.name: Xize Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:38:50.836Z
paper_authors._id: 657543227068a85089680147
paper_authors.avatarUrl: /avatars/a0e3371cc018deec7f18a12181eda3b0.svg
paper_authors.fullname: xize cheng
paper_authors.type: user
paper_authors.user: Exgc
paper_authors._id: 6712157b48e62600e97219db
paper_authors.name: Ziang Zhang
paper_authors._id: 6712157b48e62600e97219dc
paper_authors.name: Shengpeng Ji
paper_authors._id: 6712157b48e62600e97219dd
paper_authors.name: Zhou Zhao
paper_title: MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic
    Synchronization
paper_summary: Generating music that aligns with the visual content of a video has been a
  challenging task, as it requires a deep understanding of visual semantics and
  involves generating music whose melody, rhythm, and dynamics harmonize with the
  visual narratives. This paper presents MuVi, a novel framework that effectively
  addresses these challenges to enhance the cohesion and immersive experience of
  audio-visual content. MuVi analyzes video content through a specially designed
  visual adaptor to extract contextually and temporally relevant features. These
  features are used to generate music that not only matches the video's mood and
  theme but also its rhythm and pacing. We also introduce a contrastive
  music-visual pre-training scheme to ensure synchronization, based on the
  periodicity nature of music phrases. In addition, we demonstrate that our
  flow-matching-based music generator has in-context learning ability, allowing
  us to control the style and genre of the generated music. Experimental results
  show that MuVi demonstrates superior performance in both audio quality and
  temporal synchronization. The generated music video samples are available at
  https://muvi-v2m.github.io.
paper_ai_summary: MuVi, a novel framework, enhances audio-visual cohesion by generating music that matches video content's mood, theme, rhythm, and pacing using a visual adaptor and a flow-matching-based music generator.
paper_ai_keywords: visual adaptor contrastive music-visual pre-training flow-matching-based music generator in-context learning audio-visual synchronization