title: V2Meow: Meowing to the Visual Beat via Music Generation
summary: Generating high quality music that complements the visual content of a video
  is a challenging task. Most existing visual conditioned music generation
  systems generate symbolic music data, such as MIDI files, instead of raw audio
  waveform. Given the limited availability of symbolic music data, such methods
  can only generate music for a few instruments or for specific types of visual
  input. In this paper, we propose a novel approach called V2Meow that can
  generate high-quality music audio that aligns well with the visual semantics of
  a diverse range of video input types. Specifically, the proposed music
  generation system is a multi-stage autoregressive model which is trained with a
  number of O(100K) music audio clips paired with video frames, which are mined
  from in-the-wild music videos, and no parallel symbolic music data is involved.
  V2Meow is able to synthesize high-fidelity music audio waveform solely
  conditioned on pre-trained visual features extracted from an arbitrary silent
  video clip, and it also allows high-level control over the music style of
  generation examples via supporting text prompts in addition to the video frames
  conditioning. Through both qualitative and quantitative evaluations, we
  demonstrate that our model outperforms several existing music generation
  systems in terms of both visual-audio correspondence and audio quality.
paper_authors._id: 645d9bdff5760d1530d6b66a
paper_authors.name: Kun Su
paper_authors._id: 645d9bdff5760d1530d6b66b
paper_authors.name: Judith Yue Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T08:57:54.072Z
paper_authors._id: 5f431c9f79c1ba4c353d0d7f
paper_authors.avatarUrl: /avatars/7ac0936f469e63cfe8ce15849e128a43.svg
paper_authors.fullname: Judith Yue Li
paper_authors.type: user
paper_authors.user: whereisdodo
paper_authors._id: 645d9bdff5760d1530d6b66c
paper_authors.name: Qingqing Huang
paper_authors._id: 645d9bdff5760d1530d6b66d
paper_authors.name: Dima Kuzmin
paper_authors._id: 645d9bdff5760d1530d6b66e
paper_authors.name: Joonseok Lee
paper_authors._id: 645d9bdff5760d1530d6b66f
paper_authors.name: Chris Donahue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T08:58:41.505Z
paper_authors._id: 63b1b5f034b91b8272aac6c8
paper_authors.avatarUrl: /avatars/155351aa15967b5346144c03fc87d1ca.svg
paper_authors.fullname: Chris Donahue
paper_authors.type: user
paper_authors.user: chrisdonahue
paper_authors._id: 645d9bdff5760d1530d6b670
paper_authors.name: Fei Sha
paper_authors._id: 645d9bdff5760d1530d6b671
paper_authors.name: Aren Jansen
paper_authors._id: 645d9bdff5760d1530d6b672
paper_authors.name: Yu Wang
paper_authors._id: 645d9bdff5760d1530d6b673
paper_authors.name: Mauro Verzetti
paper_authors._id: 645d9bdff5760d1530d6b674
paper_authors.name: Timo I. Denk
paper_title: V2Meow: Meowing to the Visual Beat via Music Generation
paper_summary: Generating high quality music that complements the visual content of a video
  is a challenging task. Most existing visual conditioned music generation
  systems generate symbolic music data, such as MIDI files, instead of raw audio
  waveform. Given the limited availability of symbolic music data, such methods
  can only generate music for a few instruments or for specific types of visual
  input. In this paper, we propose a novel approach called V2Meow that can
  generate high-quality music audio that aligns well with the visual semantics of
  a diverse range of video input types. Specifically, the proposed music
  generation system is a multi-stage autoregressive model which is trained with a
  number of O(100K) music audio clips paired with video frames, which are mined
  from in-the-wild music videos, and no parallel symbolic music data is involved.
  V2Meow is able to synthesize high-fidelity music audio waveform solely
  conditioned on pre-trained visual features extracted from an arbitrary silent
  video clip, and it also allows high-level control over the music style of
  generation examples via supporting text prompts in addition to the video frames
  conditioning. Through both qualitative and quantitative evaluations, we
  demonstrate that our model outperforms several existing music generation
  systems in terms of both visual-audio correspondence and audio quality.
paper_ai_keywords: multi-stage autoregressive model high-quality music audio visual semantics music generation system V2Meow pre-trained visual features text prompts visual-audio correspondence audio quality