SongCreator: Lyrics-based Universal Song Generation
Music is an integral part of human culture, embodying human intelligence and
creativity, of which songs compose an essential part. While various aspects of
song generation have been explored by previous works, such as singing voice,
vocal composition and instrumental arrangement, etc., generating songs with
both vocals and accompaniment given lyrics remains a significant challenge,
hindering the application of music generation models in the real world. In this
light, we propose SongCreator, a song-generation system designed to tackle this
challenge. The model features two novel designs: a meticulously designed
dual-sequence language model (DSLM) to capture the information of vocals and
accompaniment for song generation, and an additional attention mask strategy
for DSLM, which allows our model to understand, generate and edit songs, making
it suitable for various song-related generation tasks. Extensive experiments
demonstrate the effectiveness of SongCreator by achieving state-of-the-art or
competitive performances on all eight tasks. Notably, it surpasses previous
works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally,
it is able to independently control the acoustic conditions of the vocals and
accompaniment in the generated song through different prompts, exhibiting its
potential applicability. Our samples are available at
https://songcreator.github.io/.
66e0fce94fe31d009f445b50
Shun Lei
66e0fce94fe31d009f445b51
Yixuan Zhou
66e0fce94fe31d009f445b52
Boshi Tang
66e0fce94fe31d009f445b53
Max W. Y. Lam
admin_assigned
2024-09-11T10:00:24.433Z
6278d25ae8e956767ec7e6bd
/avatars/59ce8fdee94f113b3334eea600a34ca9.svg
Max W. Y. Lam
user
maxingaussian
66e0fce94fe31d009f445b54
Feng Liu
66e0fce94fe31d009f445b55
Hangyu Liu
admin_assigned
2024-09-11T10:01:12.370Z
6616a3ace557753f30ec0e06
/avatars/4c9695be6fe34af41b6a499ec4d4b340.svg
Hangyu Liu
user
GK50
66e0fce94fe31d009f445b56
Jingcheng Wu
admin_assigned
2024-09-11T10:01:18.241Z
648a856b867cbb43ed662811
/avatars/781911a58a5d17786e70d0d8abf6d2a1.svg
wu jingcheng
user
jingchengwu
66e0fce94fe31d009f445b57
Shiyin Kang
admin_assigned
2024-09-11T10:01:23.882Z
65dda40f22d4f4074585ecc6
/avatars/b616537ad7664d6f0d24c2b6ff098098.svg
Shiyin Kang
user
kangshiyin
66e0fce94fe31d009f445b58
Zhiyong Wu
admin_assigned
2024-09-11T10:01:30.823Z
6280e830e99dccaac4bbfde5
/avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
Zhiyong Wu
user
zy001
66e0fce94fe31d009f445b59
Helen Meng
admin_assigned
2024-09-11T10:01:41.283Z
641b9791f902cc42730f43cc
/avatars/891dadc5e37bf7259d50b0c62bd8d0fb.svg
Helen M
user
helenm
SongCreator: Lyrics-based Universal Song Generation
Music is an integral part of human culture, embodying human intelligence and
creativity, of which songs compose an essential part. While various aspects of
song generation have been explored by previous works, such as singing voice,
vocal composition and instrumental arrangement, etc., generating songs with
both vocals and accompaniment given lyrics remains a significant challenge,
hindering the application of music generation models in the real world. In this
light, we propose SongCreator, a song-generation system designed to tackle this
challenge. The model features two novel designs: a meticulously designed
dual-sequence language model (DSLM) to capture the information of vocals and
accompaniment for song generation, and an additional attention mask strategy
for DSLM, which allows our model to understand, generate and edit songs, making
it suitable for various song-related generation tasks. Extensive experiments
demonstrate the effectiveness of SongCreator by achieving state-of-the-art or
competitive performances on all eight tasks. Notably, it surpasses previous
works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally,
it is able to independently control the acoustic conditions of the vocals and
accompaniment in the generated song through different prompts, exhibiting its
potential applicability. Our samples are available at
https://songcreator.github.io/.
SongCreator is a dual-sequence language model with an attention mask strategy that generates songs from lyrics, achieving state-of-the-art performance in lyrics-to-song and lyrics-to-vocals tasks while controlling acoustic conditions independently.
dual-sequence language model DSLM attention mask strategy