title: VMAS: Video-to-Music Generation via Semantic Alignment in Web Music
    Videos
summary: We present a framework for learning to generate background music from video
  inputs. Unlike existing works that rely on symbolic musical annotations, which
  are limited in quantity and diversity, our method leverages large-scale web
  videos accompanied by background music. This enables our model to learn to
  generate realistic and diverse music. To accomplish this goal, we develop a
  generative video-music Transformer with a novel semantic video-music alignment
  scheme. Our model uses a joint autoregressive and contrastive learning
  objective, which encourages the generation of music aligned with high-level
  video content. We also introduce a novel video-beat alignment scheme to match
  the generated music beats with the low-level motions in the video. Lastly, to
  capture fine-grained visual cues in a video needed for realistic background
  music generation, we introduce a new temporal video encoder architecture,
  allowing us to efficiently process videos consisting of many densely sampled
  frames. We train our framework on our newly curated DISCO-MV dataset,
  consisting of 2.2M video-music samples, which is orders of magnitude larger
  than any prior datasets used for video music generation. Our method outperforms
  existing approaches on the DISCO-MV and MusicCaps datasets according to various
  music generation evaluation metrics, including human evaluation. Results are
  available at https://genjib.github.io/project_page/VMAs/index.html
paper_authors._id: 66e25bcb5491a1f2498e5ffe
paper_authors.name: Yan-Bo Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:57.782Z
paper_authors._id: 63a715c38729ce5b5640b80b
paper_authors.avatarUrl: /avatars/5dc0a167c5dae7756a77d79cfd6334b2.svg
paper_authors.fullname: yb
paper_authors.type: user
paper_authors.user: genjib
paper_authors._id: 66e25bcb5491a1f2498e5fff
paper_authors.name: Yu Tian
paper_authors._id: 66e25bcb5491a1f2498e6000
paper_authors.name: Linjie Yang
paper_authors._id: 66e25bcb5491a1f2498e6001
paper_authors.name: Gedas Bertasius
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T08:14:06.614Z
paper_authors._id: 65d60993c2c6418adc99206a
paper_authors.avatarUrl: /avatars/fb1f9d5af7022bddf4cb090cd477108d.svg
paper_authors.fullname: Gedas Bertasius
paper_authors.type: user
paper_authors.user: gberta
paper_authors._id: 66e25bcb5491a1f2498e6002
paper_authors.name: Heng Wang
paper_title: VMAS: Video-to-Music Generation via Semantic Alignment in Web Music
    Videos
paper_summary: We present a framework for learning to generate background music from video
  inputs. Unlike existing works that rely on symbolic musical annotations, which
  are limited in quantity and diversity, our method leverages large-scale web
  videos accompanied by background music. This enables our model to learn to
  generate realistic and diverse music. To accomplish this goal, we develop a
  generative video-music Transformer with a novel semantic video-music alignment
  scheme. Our model uses a joint autoregressive and contrastive learning
  objective, which encourages the generation of music aligned with high-level
  video content. We also introduce a novel video-beat alignment scheme to match
  the generated music beats with the low-level motions in the video. Lastly, to
  capture fine-grained visual cues in a video needed for realistic background
  music generation, we introduce a new temporal video encoder architecture,
  allowing us to efficiently process videos consisting of many densely sampled
  frames. We train our framework on our newly curated DISCO-MV dataset,
  consisting of 2.2M video-music samples, which is orders of magnitude larger
  than any prior datasets used for video music generation. Our method outperforms
  existing approaches on the DISCO-MV and MusicCaps datasets according to various
  music generation evaluation metrics, including human evaluation. Results are
  available at https://genjib.github.io/project_page/VMAs/index.html
paper_ai_keywords: generative video-music Transformer semantic video-music alignment joint autoregressive and contrastive learning video-beat alignment temporal video encoder architecture DISCO-MV dataset video music generation human evaluation