JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning
<|reference_start|>JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning: Large models for text-to-music generation have achieved significant progress, facilitating the creation of high-quality and varied musical compositions from provided text prompts. However, input text prompts may not precisely capture user requirements, particularly when the objective is to generate music that embodies a specific concept derived from a designated reference collection. In this paper, we propose a novel method for customized text-to-music generation, which can capture the concept from a two-minute reference music and generate a new piece of music conforming to the concept. We achieve this by fine-tuning a pretrained text-to-music model using the reference music. However, directly fine-tuning all parameters leads to overfitting issues. To address this problem, we propose a Pivotal Parameters Tuning method that enables the model to assimilate the new concept while preserving its original generative capabilities. Additionally, we identify a potential concept conflict when introducing multiple concepts into the pretrained model. We present a concept enhancement strategy to distinguish multiple concepts, enabling the fine-tuned model to generate music incorporating either individual or multiple concepts simultaneously. Since we are the first to work on the customized music generation task, we also introduce a new dataset and evaluation protocol for the new task. Our proposed Jen1-DreamStyler outperforms several baselines in both qualitative and quantitative evaluations. Demos will be available at https://www.jenmusic.ai/research#DreamStyler.<|reference_end|>
@article{chen2024jen-1,
    title={JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal
  Parameters Tuning},
    author={Boyu Chen, Peike Li, Yao Yao, Alex Wang},
    journal={arXiv preprint arXiv:2406.12292},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12292},
    primaryClass={cs.SD cs.AI eess.AS}
}