title: JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal
    Parameters Tuning
summary: Large models for text-to-music generation have achieved significant progress,
  facilitating the creation of high-quality and varied musical compositions from
  provided text prompts. However, input text prompts may not precisely capture
  user requirements, particularly when the objective is to generate music that
  embodies a specific concept derived from a designated reference collection. In
  this paper, we propose a novel method for customized text-to-music generation,
  which can capture the concept from a two-minute reference music and generate a
  new piece of music conforming to the concept. We achieve this by fine-tuning a
  pretrained text-to-music model using the reference music. However, directly
  fine-tuning all parameters leads to overfitting issues. To address this
  problem, we propose a Pivotal Parameters Tuning method that enables the model
  to assimilate the new concept while preserving its original generative
  capabilities. Additionally, we identify a potential concept conflict when
  introducing multiple concepts into the pretrained model. We present a concept
  enhancement strategy to distinguish multiple concepts, enabling the fine-tuned
  model to generate music incorporating either individual or multiple concepts
  simultaneously. Since we are the first to work on the customized music
  generation task, we also introduce a new dataset and evaluation protocol for
  the new task. Our proposed Jen1-DreamStyler outperforms several baselines in
  both qualitative and quantitative evaluations. Demos will be available at
  https://www.jenmusic.ai/research#DreamStyler.
paper_authors._id: 6672597e7e7ff5d8bd0b18cf
paper_authors.name: Boyu Chen
paper_authors._id: 6672597e7e7ff5d8bd0b18d0
paper_authors.name: Peike Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:11:00.982Z
paper_authors._id: 6367054f361a96184dc44db0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667695944853-noauth.png
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: Peike
paper_authors._id: 6672597e7e7ff5d8bd0b18d1
paper_authors.name: Yao Yao
paper_authors._id: 6672597e7e7ff5d8bd0b18d2
paper_authors.name: Alex Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:34:27.805Z
paper_authors._id: 6673cbc4ccd71b5bb7609a62
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/M543BjPm3K4A-OBYOFUEE.png
paper_authors.fullname: Dr. Alex Wang
paper_authors.type: user
paper_authors.user: dralexw
paper_title: JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal
    Parameters Tuning
paper_summary: Large models for text-to-music generation have achieved significant progress,
  facilitating the creation of high-quality and varied musical compositions from
  provided text prompts. However, input text prompts may not precisely capture
  user requirements, particularly when the objective is to generate music that
  embodies a specific concept derived from a designated reference collection. In
  this paper, we propose a novel method for customized text-to-music generation,
  which can capture the concept from a two-minute reference music and generate a
  new piece of music conforming to the concept. We achieve this by fine-tuning a
  pretrained text-to-music model using the reference music. However, directly
  fine-tuning all parameters leads to overfitting issues. To address this
  problem, we propose a Pivotal Parameters Tuning method that enables the model
  to assimilate the new concept while preserving its original generative
  capabilities. Additionally, we identify a potential concept conflict when
  introducing multiple concepts into the pretrained model. We present a concept
  enhancement strategy to distinguish multiple concepts, enabling the fine-tuned
  model to generate music incorporating either individual or multiple concepts
  simultaneously. Since we are the first to work on the customized music
  generation task, we also introduce a new dataset and evaluation protocol for
  the new task. Our proposed Jen1-DreamStyler outperforms several baselines in
  both qualitative and quantitative evaluations. Demos will be available at
  https://www.jenmusic.ai/research#DreamStyler.
paper_ai_summary: A novel method for customized text-to-music generation using pivotal parameters tuning and concept enhancement outperforms baselines in qualitative and quantitative evaluations.
paper_ai_keywords: text-to-music generation pretrained model fine-tuning overfitting pivotal parameters tuning concept conflict concept enhancement new dataset evaluation protocol Jen1-DreamStyler