title: SongCreator: Lyrics-based Universal Song Generation
summary: Music is an integral part of human culture, embodying human intelligence and
  creativity, of which songs compose an essential part. While various aspects of
  song generation have been explored by previous works, such as singing voice,
  vocal composition and instrumental arrangement, etc., generating songs with
  both vocals and accompaniment given lyrics remains a significant challenge,
  hindering the application of music generation models in the real world. In this
  light, we propose SongCreator, a song-generation system designed to tackle this
  challenge. The model features two novel designs: a meticulously designed
  dual-sequence language model (DSLM) to capture the information of vocals and
  accompaniment for song generation, and an additional attention mask strategy
  for DSLM, which allows our model to understand, generate and edit songs, making
  it suitable for various song-related generation tasks. Extensive experiments
  demonstrate the effectiveness of SongCreator by achieving state-of-the-art or
  competitive performances on all eight tasks. Notably, it surpasses previous
  works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally,
  it is able to independently control the acoustic conditions of the vocals and
  accompaniment in the generated song through different prompts, exhibiting its
  potential applicability. Our samples are available at
  https://songcreator.github.io/.
paper_authors._id: 66e0fce94fe31d009f445b50
paper_authors.name: Shun Lei
paper_authors._id: 66e0fce94fe31d009f445b51
paper_authors.name: Yixuan Zhou
paper_authors._id: 66e0fce94fe31d009f445b52
paper_authors.name: Boshi Tang
paper_authors._id: 66e0fce94fe31d009f445b53
paper_authors.name: Max W. Y. Lam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:00:24.433Z
paper_authors._id: 6278d25ae8e956767ec7e6bd
paper_authors.avatarUrl: /avatars/59ce8fdee94f113b3334eea600a34ca9.svg
paper_authors.fullname: Max W. Y. Lam
paper_authors.type: user
paper_authors.user: maxingaussian
paper_authors._id: 66e0fce94fe31d009f445b54
paper_authors.name: Feng Liu
paper_authors._id: 66e0fce94fe31d009f445b55
paper_authors.name: Hangyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:01:12.370Z
paper_authors._id: 6616a3ace557753f30ec0e06
paper_authors.avatarUrl: /avatars/4c9695be6fe34af41b6a499ec4d4b340.svg
paper_authors.fullname: Hangyu Liu
paper_authors.type: user
paper_authors.user: GK50
paper_authors._id: 66e0fce94fe31d009f445b56
paper_authors.name: Jingcheng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:01:18.241Z
paper_authors._id: 648a856b867cbb43ed662811
paper_authors.avatarUrl: /avatars/781911a58a5d17786e70d0d8abf6d2a1.svg
paper_authors.fullname: wu jingcheng
paper_authors.type: user
paper_authors.user: jingchengwu
paper_authors._id: 66e0fce94fe31d009f445b57
paper_authors.name: Shiyin Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:01:23.882Z
paper_authors._id: 65dda40f22d4f4074585ecc6
paper_authors.avatarUrl: /avatars/b616537ad7664d6f0d24c2b6ff098098.svg
paper_authors.fullname: Shiyin Kang
paper_authors.type: user
paper_authors.user: kangshiyin
paper_authors._id: 66e0fce94fe31d009f445b58
paper_authors.name: Zhiyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:01:30.823Z
paper_authors._id: 6280e830e99dccaac4bbfde5
paper_authors.avatarUrl: /avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
paper_authors.fullname: Zhiyong Wu
paper_authors.type: user
paper_authors.user: zy001
paper_authors._id: 66e0fce94fe31d009f445b59
paper_authors.name: Helen Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:01:41.283Z
paper_authors._id: 641b9791f902cc42730f43cc
paper_authors.avatarUrl: /avatars/891dadc5e37bf7259d50b0c62bd8d0fb.svg
paper_authors.fullname: Helen M
paper_authors.type: user
paper_authors.user: helenm
paper_title: SongCreator: Lyrics-based Universal Song Generation
paper_summary: Music is an integral part of human culture, embodying human intelligence and
  creativity, of which songs compose an essential part. While various aspects of
  song generation have been explored by previous works, such as singing voice,
  vocal composition and instrumental arrangement, etc., generating songs with
  both vocals and accompaniment given lyrics remains a significant challenge,
  hindering the application of music generation models in the real world. In this
  light, we propose SongCreator, a song-generation system designed to tackle this
  challenge. The model features two novel designs: a meticulously designed
  dual-sequence language model (DSLM) to capture the information of vocals and
  accompaniment for song generation, and an additional attention mask strategy
  for DSLM, which allows our model to understand, generate and edit songs, making
  it suitable for various song-related generation tasks. Extensive experiments
  demonstrate the effectiveness of SongCreator by achieving state-of-the-art or
  competitive performances on all eight tasks. Notably, it surpasses previous
  works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally,
  it is able to independently control the acoustic conditions of the vocals and
  accompaniment in the generated song through different prompts, exhibiting its
  potential applicability. Our samples are available at
  https://songcreator.github.io/.
paper_ai_summary: SongCreator is a dual-sequence language model with an attention mask strategy that generates songs from lyrics, achieving state-of-the-art performance in lyrics-to-song and lyrics-to-vocals tasks while controlling acoustic conditions independently.
paper_ai_keywords: dual-sequence language model DSLM attention mask strategy