Foundation Models for Music: A Survey
In recent years, foundation models (FMs) such as large language models (LLMs)
and latent diffusion models (LDMs) have profoundly impacted diverse sectors,
including music. This comprehensive review examines state-of-the-art (SOTA)
pre-trained models and foundation models in music, spanning from representation
learning, generative learning and multimodal learning. We first contextualise
the significance of music in various industries and trace the evolution of AI
in music. By delineating the modalities targeted by foundation models, we
discover many of the music representations are underexplored in FM development.
Then, emphasis is placed on the lack of versatility of previous methods on
diverse music applications, along with the potential of FMs in music
understanding, generation and medical application. By comprehensively exploring
the details of the model pre-training paradigm, architectural choices,
tokenisation, finetuning methodologies and controllability, we emphasise the
important topics that should have been well explored, like instruction tuning
and in-context learning, scaling law and emergent ability, as well as
long-sequence modelling etc. A dedicated section presents insights into music
agents, accompanied by a thorough analysis of datasets and evaluations
essential for pre-training and downstream tasks. Finally, by underscoring the
vital importance of ethical considerations, we advocate that following research
on FM for music should focus more on such issues as interpretability,
transparency, human responsibility, and copyright issues. The paper offers
insights into future challenges and trends on FMs for music, aiming to shape
the trajectory of human-AI collaboration in the music realm.
66cd4bff2299fbc852eb156a
Yinghao Ma
admin_assigned
2024-08-27T12:26:39.164Z
6410665d5364a661bee22524
/avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
Yinghao Ma
user
nicolaus625
66cd4bff2299fbc852eb156b
Anders Øland
admin_assigned
2024-08-27T12:26:59.955Z
65dba61e7be5025e67fe1cbd
/avatars/18995377a9a62da527fc4217a7db0282.svg
Anders Oland
user
andersoland
66cd4bff2299fbc852eb156c
Anton Ragni
66cd4bff2299fbc852eb156d
Bleiz MacSen Del Sette
66cd4bff2299fbc852eb156e
Charalampos Saitis
66cd4bff2299fbc852eb156f
Chris Donahue
admin_assigned
2024-08-27T12:27:20.540Z
63b1b5f034b91b8272aac6c8
/avatars/155351aa15967b5346144c03fc87d1ca.svg
Chris Donahue
user
chrisdonahue
66cd4bff2299fbc852eb1570
Chenghua Lin
admin_assigned
2024-08-27T12:27:26.403Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
66cd4bff2299fbc852eb1571
Christos Plachouras
admin_assigned
2024-08-27T12:27:35.079Z
62ff90a499abacb6f2975b84
https://cdn-avatars.huggingface.co/v1/production/uploads/62ff90a499abacb6f2975b84/wrrC8vVdj1Mhg1jpODA2-.jpeg
Christos Plachouras
user
chrispla
66cd4bff2299fbc852eb1572
Emmanouil Benetos
admin_assigned
2024-08-27T12:27:40.386Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
66cd4bff2299fbc852eb1573
Elio Quinton
admin_assigned
2024-08-27T12:27:45.985Z
64f1c4bf4fc14420f6935e7f
/avatars/b01e80f10d0de3f6b0da8b617299d9b7.svg
Elio Quinton
user
elio-quinton-umg
66cd4bff2299fbc852eb1574
Elona Shatri
admin_assigned
2024-08-27T12:27:51.016Z
6357f6bd11f8b111de48450f
/avatars/6db2ff667804d42a0f00e03fc2636fda.svg
Elona Shatri
user
penestia
66cd4bff2299fbc852eb1575
Fabio Morreale
admin_assigned
2024-08-27T12:27:56.433Z
648687555bc2107499b97d58
https://cdn-avatars.huggingface.co/v1/production/uploads/648687555bc2107499b97d58/vCd1d5hW8-7BhxuwlSKVY.jpeg
Fabio Morreale
user
fmorr
66cd4bff2299fbc852eb1576
Ge Zhang
claimed_verified
2024-08-27T07:22:42.255Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66cd4bff2299fbc852eb1577
György Fazekas
66cd4bff2299fbc852eb1578
Gus Xia
66cd4bff2299fbc852eb1579
Huan Zhang
admin_assigned
2024-08-27T12:28:16.251Z
64c286b4b32fc3f9e9525a02
/avatars/7ce42d920ded4aa7033abe2fb7245edd.svg
Huan Zhang
user
anusfoil
66cd4bff2299fbc852eb157a
Ilaria Manco
admin_assigned
2024-08-27T12:28:21.419Z
64ed060f33d50752497d4a16
https://cdn-avatars.huggingface.co/v1/production/uploads/64ed060f33d50752497d4a16/0QbK9jeGvY3DBMVHDdTDX.jpeg
Ilaria Manco
user
ilaria-manco
66cd4bff2299fbc852eb157b
Jiawen Huang
66cd4bff2299fbc852eb157c
Julien Guinot
66cd4bff2299fbc852eb157d
Liwei Lin
admin_assigned
2024-08-27T12:29:11.373Z
66324a8cfebe847d074da206
/avatars/a9498b2fb36c4508dacbf61a4458e2dd.svg
Liwei Lin
user
Kikyo-16
66cd4bff2299fbc852eb157e
Luca Marinelli
66cd4bff2299fbc852eb157f
Max W. Y. Lam
66cd4bff2299fbc852eb1580
Megha Sharma
66cd4bff2299fbc852eb1581
Qiuqiang Kong
66cd4bff2299fbc852eb1582
Roger B. Dannenberg
66cd4bff2299fbc852eb1583
Ruibin Yuan
claimed_verified
2025-03-12T08:53:53.863Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
66cd4bff2299fbc852eb1584
Shangda Wu
66cd4bff2299fbc852eb1585
Shih-Lun Wu
66cd4bff2299fbc852eb1586
Shuqi Dai
66cd4bff2299fbc852eb1587
Shun Lei
66cd4bff2299fbc852eb1588
Shiyin Kang
66cd4bff2299fbc852eb1589
Simon Dixon
66cd4bff2299fbc852eb158a
Wenhu Chen
66cd4bff2299fbc852eb158b
Wehhao Huang
claimed_verified
2024-10-13T20:25:47.473Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
66cd4bff2299fbc852eb158c
Xingjian Du
66cd4bff2299fbc852eb158d
Xingwei Qu
66cd4bff2299fbc852eb158e
Xu Tan
claimed_verified
2024-08-28T13:47:09.935Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
66cd4bff2299fbc852eb158f
Yizhi Li
claimed_verified
2025-07-25T08:40:28.682Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
66cd4bff2299fbc852eb1590
Zeyue Tian
claimed_verified
2025-03-16T21:23:54.181Z
6504409b52ca06fef957266d
/avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
Zeyue
user
Zeyue7
66cd4bff2299fbc852eb1591
Zhiyong Wu
66cd4bff2299fbc852eb1592
Zhizheng Wu
66cd4bff2299fbc852eb1593
Ziyang Ma
66cd4bff2299fbc852eb1594
Ziyu Wang
claimed_verified
2024-10-18T10:32:56.237Z
666e9aef15b60b3fba91fce1
/avatars/2b27cc8895e6f21c02b751791a568962.svg
Ziyu Wang
user
ZzWaang
Foundation Models for Music: A Survey
In recent years, foundation models (FMs) such as large language models (LLMs)
and latent diffusion models (LDMs) have profoundly impacted diverse sectors,
including music. This comprehensive review examines state-of-the-art (SOTA)
pre-trained models and foundation models in music, spanning from representation
learning, generative learning and multimodal learning. We first contextualise
the significance of music in various industries and trace the evolution of AI
in music. By delineating the modalities targeted by foundation models, we
discover many of the music representations are underexplored in FM development.
Then, emphasis is placed on the lack of versatility of previous methods on
diverse music applications, along with the potential of FMs in music
understanding, generation and medical application. By comprehensively exploring
the details of the model pre-training paradigm, architectural choices,
tokenisation, finetuning methodologies and controllability, we emphasise the
important topics that should have been well explored, like instruction tuning
and in-context learning, scaling law and emergent ability, as well as
long-sequence modelling etc. A dedicated section presents insights into music
agents, accompanied by a thorough analysis of datasets and evaluations
essential for pre-training and downstream tasks. Finally, by underscoring the
vital importance of ethical considerations, we advocate that following research
on FM for music should focus more on such issues as interpretability,
transparency, human responsibility, and copyright issues. The paper offers
insights into future challenges and trends on FMs for music, aiming to shape
the trajectory of human-AI collaboration in the music realm.
A review of foundation models in music, including large language models and latent diffusion models, highlights their impact, architectural choices, and the need for ethical considerations in music applications.
large language models latent diffusion models representation learning generative learning multimodal learning instruction tuning in-context learning scaling law emergent ability long-sequence modelling music agents