Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language
  Models via Instruction Tuning
Recent advances in text-to-music editing, which employ text queries to modify
music (e.g.\ by changing its style or adjusting instrumental components),
present unique challenges and opportunities for AI-assisted music creation.
Previous approaches in this domain have been constrained by the necessity to
train specific editing models from scratch, which is both resource-intensive
and inefficient; other research uses large language models to predict edited
music, resulting in imprecise audio reconstruction. To Combine the strengths
and address these limitations, we introduce Instruct-MusicGen, a novel approach
that finetunes a pretrained MusicGen model to efficiently follow editing
instructions such as adding, removing, or separating stems. Our approach
involves a modification of the original MusicGen architecture by incorporating
a text fusion module and an audio fusion module, which allow the model to
process instruction texts and audio inputs concurrently and yield the desired
edited music. Remarkably, Instruct-MusicGen only introduces 8% new parameters
to the original MusicGen model and only trains for 5K steps, yet it achieves
superior performance across all tasks compared to existing baselines, and
demonstrates performance comparable to the models trained for specific tasks.
This advancement not only enhances the efficiency of text-to-music editing but
also broadens the applicability of music language models in dynamic music
production environments.
6656afa1b7e6a9f09553e291
Yixiao Zhang
admin_assigned
2024-05-29T08:23:39.716Z
6376a5aad49217e154f580aa
https://cdn-avatars.huggingface.co/v1/production/uploads/1668720030380-noauth.jpeg
Yixiao ZHang
user
ldzhangyx
6656afa1b7e6a9f09553e292
Yukara Ikemiya
admin_assigned
2024-05-29T08:23:45.761Z
62fbbdcd21c444a56f7a6159
/avatars/3f83effd52773f0806f58331d8734861.svg
Yukara Ikemiya
user
yukara13
6656afa1b7e6a9f09553e293
Woosung Choi
admin_assigned
2024-05-29T08:24:36.131Z
62c5411c9f9e408c41073c3e
/avatars/2e046d996af8441127a38ba23a6248e8.svg
Woosung Choi
user
Woosung
6656afa1b7e6a9f09553e294
Naoki Murata
admin_assigned
2024-05-29T08:24:42.188Z
63e1e9627fbb6ae4d4f59d76
/avatars/22197a8c85232135609331766299c3e7.svg
Naoki Murata
user
muttyjp
6656afa1b7e6a9f09553e295
Marco A. Martínez-Ramírez
6656afa1b7e6a9f09553e296
Liwei Lin
admin_assigned
2024-05-29T08:25:15.656Z
66324a8cfebe847d074da206
/avatars/a9498b2fb36c4508dacbf61a4458e2dd.svg
Liwei Lin
user
Kikyo-16
6656afa1b7e6a9f09553e297
Gus Xia
6656afa1b7e6a9f09553e298
Wei-Hsiang Liao
6656afa1b7e6a9f09553e299
Yuki Mitsufuji
6656afa1b7e6a9f09553e29a
Simon Dixon
admin_assigned
2024-05-29T08:26:01.380Z
643ac28aff50448bcfc24b19
/avatars/18d201b94829bae9985dbb4ec618e1ff.svg
Simon Dixon
user
Dexdixon
Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language
  Models via Instruction Tuning
Recent advances in text-to-music editing, which employ text queries to modify
music (e.g.\ by changing its style or adjusting instrumental components),
present unique challenges and opportunities for AI-assisted music creation.
Previous approaches in this domain have been constrained by the necessity to
train specific editing models from scratch, which is both resource-intensive
and inefficient; other research uses large language models to predict edited
music, resulting in imprecise audio reconstruction. To Combine the strengths
and address these limitations, we introduce Instruct-MusicGen, a novel approach
that finetunes a pretrained MusicGen model to efficiently follow editing
instructions such as adding, removing, or separating stems. Our approach
involves a modification of the original MusicGen architecture by incorporating
a text fusion module and an audio fusion module, which allow the model to
process instruction texts and audio inputs concurrently and yield the desired
edited music. Remarkably, Instruct-MusicGen only introduces 8% new parameters
to the original MusicGen model and only trains for 5K steps, yet it achieves
superior performance across all tasks compared to existing baselines, and
demonstrates performance comparable to the models trained for specific tasks.
This advancement not only enhances the efficiency of text-to-music editing but
also broadens the applicability of music language models in dynamic music
production environments.
Instruct-MusicGen, a finetuned MusicGen model with integrated text and audio modules, efficiently follows editing instructions with minimal additional parameters and achieves superior performance in text-to-music editing.
MusicGen text fusion module audio fusion module text-to-music editing parameter-efficient fine-tuning editing instructions task-specific training audio reconstruction dynamic music production