title: Language-Guided Music Recommendation for Video via Prompt Analogies
summary: We propose a method to recommend music for an input video while allowing a
  user to guide music selection with free-form natural language. A key challenge
  of this problem setting is that existing music video datasets provide the
  needed (video, music) training pairs, but lack text descriptions of the music.
  This work addresses this challenge with the following three contributions.
  First, we propose a text-synthesis approach that relies on an analogy-based
  prompting procedure to generate natural language music descriptions from a
  large-scale language model (BLOOM-176B) given pre-trained music tagger outputs
  and a small number of human text descriptions. Second, we use these synthesized
  music descriptions to train a new trimodal model, which fuses text and video
  input representations to query music samples. For training, we introduce a text
  dropout regularization mechanism which we show is critical to model
  performance. Our model design allows for the retrieved music audio to agree
  with the two input modalities by matching visual style depicted in the video
  and musical genre, mood, or instrumentation described in the natural language
  query. Third, to evaluate our approach, we collect a testing dataset for our
  problem by annotating a subset of 4k clips from the YT8M-MusicVideo dataset
  with natural language music descriptions which we make publicly available. We
  show that our approach can match or exceed the performance of prior methods on
  video-to-music retrieval while significantly improving retrieval accuracy when
  using text guidance.
paper_authors._id: 648be987335645aa03ede32d
paper_authors.name: Daniel McKee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-16T16:34:12.968Z
paper_authors._id: 62d837259d38820db26010e5
paper_authors.avatarUrl: /avatars/29cf7ec18f500349ca6c3654e195ab32.svg
paper_authors.fullname: Daniel McKee
paper_authors.type: user
paper_authors.user: dmckee5
paper_authors._id: 648be987335645aa03ede32e
paper_authors.name: Justin Salamon
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-21T22:07:37.581Z
paper_authors._id: 6418d763f9d6f1d772eb920b
paper_authors.avatarUrl: /avatars/a762a03b753a7ad1839b4074221cfab6.svg
paper_authors.fullname: Justin Salamon
paper_authors.type: user
paper_authors.user: salamon
paper_authors._id: 648be987335645aa03ede32f
paper_authors.name: Josef Sivic
paper_authors._id: 648be987335645aa03ede330
paper_authors.name: Bryan Russell
paper_title: Language-Guided Music Recommendation for Video via Prompt Analogies
paper_summary: We propose a method to recommend music for an input video while allowing a
  user to guide music selection with free-form natural language. A key challenge
  of this problem setting is that existing music video datasets provide the
  needed (video, music) training pairs, but lack text descriptions of the music.
  This work addresses this challenge with the following three contributions.
  First, we propose a text-synthesis approach that relies on an analogy-based
  prompting procedure to generate natural language music descriptions from a
  large-scale language model (BLOOM-176B) given pre-trained music tagger outputs
  and a small number of human text descriptions. Second, we use these synthesized
  music descriptions to train a new trimodal model, which fuses text and video
  input representations to query music samples. For training, we introduce a text
  dropout regularization mechanism which we show is critical to model
  performance. Our model design allows for the retrieved music audio to agree
  with the two input modalities by matching visual style depicted in the video
  and musical genre, mood, or instrumentation described in the natural language
  query. Third, to evaluate our approach, we collect a testing dataset for our
  problem by annotating a subset of 4k clips from the YT8M-MusicVideo dataset
  with natural language music descriptions which we make publicly available. We
  show that our approach can match or exceed the performance of prior methods on
  video-to-music retrieval while significantly improving retrieval accuracy when
  using text guidance.
paper_ai_keywords: text-synthesis approach analogy-based prompting procedure BLOOM-176B pre-trained music tagger outputs trimodal model text and video input representations text dropout regularization mechanism visual style depicted in the video musical genre, mood, or instrumentation video-to-music retrieval