Language-Guided Music Recommendation for Video via Prompt Analogies
We propose a method to recommend music for an input video while allowing a
user to guide music selection with free-form natural language. A key challenge
of this problem setting is that existing music video datasets provide the
needed (video, music) training pairs, but lack text descriptions of the music.
This work addresses this challenge with the following three contributions.
First, we propose a text-synthesis approach that relies on an analogy-based
prompting procedure to generate natural language music descriptions from a
large-scale language model (BLOOM-176B) given pre-trained music tagger outputs
and a small number of human text descriptions. Second, we use these synthesized
music descriptions to train a new trimodal model, which fuses text and video
input representations to query music samples. For training, we introduce a text
dropout regularization mechanism which we show is critical to model
performance. Our model design allows for the retrieved music audio to agree
with the two input modalities by matching visual style depicted in the video
and musical genre, mood, or instrumentation described in the natural language
query. Third, to evaluate our approach, we collect a testing dataset for our
problem by annotating a subset of 4k clips from the YT8M-MusicVideo dataset
with natural language music descriptions which we make publicly available. We
show that our approach can match or exceed the performance of prior methods on
video-to-music retrieval while significantly improving retrieval accuracy when
using text guidance.
648be987335645aa03ede32d
Daniel McKee
extracted_confirmed
2023-06-16T16:34:12.968Z
62d837259d38820db26010e5
/avatars/29cf7ec18f500349ca6c3654e195ab32.svg
Daniel McKee
user
dmckee5
648be987335645aa03ede32e
Justin Salamon
extracted_confirmed
2023-06-21T22:07:37.581Z
6418d763f9d6f1d772eb920b
/avatars/a762a03b753a7ad1839b4074221cfab6.svg
Justin Salamon
user
salamon
648be987335645aa03ede32f
Josef Sivic
648be987335645aa03ede330
Bryan Russell
Language-Guided Music Recommendation for Video via Prompt Analogies
We propose a method to recommend music for an input video while allowing a
user to guide music selection with free-form natural language. A key challenge
of this problem setting is that existing music video datasets provide the
needed (video, music) training pairs, but lack text descriptions of the music.
This work addresses this challenge with the following three contributions.
First, we propose a text-synthesis approach that relies on an analogy-based
prompting procedure to generate natural language music descriptions from a
large-scale language model (BLOOM-176B) given pre-trained music tagger outputs
and a small number of human text descriptions. Second, we use these synthesized
music descriptions to train a new trimodal model, which fuses text and video
input representations to query music samples. For training, we introduce a text
dropout regularization mechanism which we show is critical to model
performance. Our model design allows for the retrieved music audio to agree
with the two input modalities by matching visual style depicted in the video
and musical genre, mood, or instrumentation described in the natural language
query. Third, to evaluate our approach, we collect a testing dataset for our
problem by annotating a subset of 4k clips from the YT8M-MusicVideo dataset
with natural language music descriptions which we make publicly available. We
show that our approach can match or exceed the performance of prior methods on
video-to-music retrieval while significantly improving retrieval accuracy when
using text guidance.
A text-synthesis approach using a large-scale language model and a new trimodal model enhances video-to-music recommendation with natural language inputs, outperforming prior methods in retrieval accuracy.
text-synthesis analogy-based prompting BLOOM-176B music tagger trimodal model text dropout regularization video-to-music retrieval