title: Foundation Models for Music: A Survey
summary: In recent years, foundation models (FMs) such as large language models (LLMs)
  and latent diffusion models (LDMs) have profoundly impacted diverse sectors,
  including music. This comprehensive review examines state-of-the-art (SOTA)
  pre-trained models and foundation models in music, spanning from representation
  learning, generative learning and multimodal learning. We first contextualise
  the significance of music in various industries and trace the evolution of AI
  in music. By delineating the modalities targeted by foundation models, we
  discover many of the music representations are underexplored in FM development.
  Then, emphasis is placed on the lack of versatility of previous methods on
  diverse music applications, along with the potential of FMs in music
  understanding, generation and medical application. By comprehensively exploring
  the details of the model pre-training paradigm, architectural choices,
  tokenisation, finetuning methodologies and controllability, we emphasise the
  important topics that should have been well explored, like instruction tuning
  and in-context learning, scaling law and emergent ability, as well as
  long-sequence modelling etc. A dedicated section presents insights into music
  agents, accompanied by a thorough analysis of datasets and evaluations
  essential for pre-training and downstream tasks. Finally, by underscoring the
  vital importance of ethical considerations, we advocate that following research
  on FM for music should focus more on such issues as interpretability,
  transparency, human responsibility, and copyright issues. The paper offers
  insights into future challenges and trends on FMs for music, aiming to shape
  the trajectory of human-AI collaboration in the music realm.
paper_authors._id: 66cd4bff2299fbc852eb156a
paper_authors.name: Yinghao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:26:39.164Z
paper_authors._id: 6410665d5364a661bee22524
paper_authors.avatarUrl: /avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
paper_authors.fullname: Yinghao Ma
paper_authors.type: user
paper_authors.user: nicolaus625
paper_authors._id: 66cd4bff2299fbc852eb156b
paper_authors.name: Anders Øland
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:26:59.955Z
paper_authors._id: 65dba61e7be5025e67fe1cbd
paper_authors.avatarUrl: /avatars/18995377a9a62da527fc4217a7db0282.svg
paper_authors.fullname: Anders Oland
paper_authors.type: user
paper_authors.user: andersoland
paper_authors._id: 66cd4bff2299fbc852eb156c
paper_authors.name: Anton Ragni
paper_authors._id: 66cd4bff2299fbc852eb156d
paper_authors.name: Bleiz MacSen Del Sette
paper_authors._id: 66cd4bff2299fbc852eb156e
paper_authors.name: Charalampos Saitis
paper_authors._id: 66cd4bff2299fbc852eb156f
paper_authors.name: Chris Donahue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:20.540Z
paper_authors._id: 63b1b5f034b91b8272aac6c8
paper_authors.avatarUrl: /avatars/155351aa15967b5346144c03fc87d1ca.svg
paper_authors.fullname: Chris Donahue
paper_authors.type: user
paper_authors.user: chrisdonahue
paper_authors._id: 66cd4bff2299fbc852eb1570
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:26.403Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_authors._id: 66cd4bff2299fbc852eb1571
paper_authors.name: Christos Plachouras
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:35.079Z
paper_authors._id: 62ff90a499abacb6f2975b84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ff90a499abacb6f2975b84/wrrC8vVdj1Mhg1jpODA2-.jpeg
paper_authors.fullname: Christos Plachouras
paper_authors.type: user
paper_authors.user: chrispla
paper_authors._id: 66cd4bff2299fbc852eb1572
paper_authors.name: Emmanouil Benetos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:40.386Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 66cd4bff2299fbc852eb1573
paper_authors.name: Elio Quinton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:45.985Z
paper_authors._id: 64f1c4bf4fc14420f6935e7f
paper_authors.avatarUrl: /avatars/b01e80f10d0de3f6b0da8b617299d9b7.svg
paper_authors.fullname: Elio Quinton
paper_authors.type: user
paper_authors.user: elio-quinton-umg
paper_authors._id: 66cd4bff2299fbc852eb1574
paper_authors.name: Elona Shatri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:51.016Z
paper_authors._id: 6357f6bd11f8b111de48450f
paper_authors.avatarUrl: /avatars/6db2ff667804d42a0f00e03fc2636fda.svg
paper_authors.fullname: Elona Shatri
paper_authors.type: user
paper_authors.user: penestia
paper_authors._id: 66cd4bff2299fbc852eb1575
paper_authors.name: Fabio Morreale
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:27:56.433Z
paper_authors._id: 648687555bc2107499b97d58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648687555bc2107499b97d58/vCd1d5hW8-7BhxuwlSKVY.jpeg
paper_authors.fullname: Fabio Morreale
paper_authors.type: user
paper_authors.user: fmorr
paper_authors._id: 66cd4bff2299fbc852eb1576
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:22:42.255Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66cd4bff2299fbc852eb1577
paper_authors.name: György Fazekas
paper_authors._id: 66cd4bff2299fbc852eb1578
paper_authors.name: Gus Xia
paper_authors._id: 66cd4bff2299fbc852eb1579
paper_authors.name: Huan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:28:16.251Z
paper_authors._id: 64c286b4b32fc3f9e9525a02
paper_authors.avatarUrl: /avatars/7ce42d920ded4aa7033abe2fb7245edd.svg
paper_authors.fullname: Huan Zhang
paper_authors.type: user
paper_authors.user: anusfoil
paper_authors._id: 66cd4bff2299fbc852eb157a
paper_authors.name: Ilaria Manco
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:28:21.419Z
paper_authors._id: 64ed060f33d50752497d4a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ed060f33d50752497d4a16/0QbK9jeGvY3DBMVHDdTDX.jpeg
paper_authors.fullname: Ilaria Manco
paper_authors.type: user
paper_authors.user: ilaria-manco
paper_authors._id: 66cd4bff2299fbc852eb157b
paper_authors.name: Jiawen Huang
paper_authors._id: 66cd4bff2299fbc852eb157c
paper_authors.name: Julien Guinot
paper_authors._id: 66cd4bff2299fbc852eb157d
paper_authors.name: Liwei Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:29:11.373Z
paper_authors._id: 66324a8cfebe847d074da206
paper_authors.avatarUrl: /avatars/a9498b2fb36c4508dacbf61a4458e2dd.svg
paper_authors.fullname: Liwei Lin
paper_authors.type: user
paper_authors.user: Kikyo-16
paper_authors._id: 66cd4bff2299fbc852eb157e
paper_authors.name: Luca Marinelli
paper_authors._id: 66cd4bff2299fbc852eb157f
paper_authors.name: Max W. Y. Lam
paper_authors._id: 66cd4bff2299fbc852eb1580
paper_authors.name: Megha Sharma
paper_authors._id: 66cd4bff2299fbc852eb1581
paper_authors.name: Qiuqiang Kong
paper_authors._id: 66cd4bff2299fbc852eb1582
paper_authors.name: Roger B. Dannenberg
paper_authors._id: 66cd4bff2299fbc852eb1583
paper_authors.name: Ruibin Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:53:53.863Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 66cd4bff2299fbc852eb1584
paper_authors.name: Shangda Wu
paper_authors._id: 66cd4bff2299fbc852eb1585
paper_authors.name: Shih-Lun Wu
paper_authors._id: 66cd4bff2299fbc852eb1586
paper_authors.name: Shuqi Dai
paper_authors._id: 66cd4bff2299fbc852eb1587
paper_authors.name: Shun Lei
paper_authors._id: 66cd4bff2299fbc852eb1588
paper_authors.name: Shiyin Kang
paper_authors._id: 66cd4bff2299fbc852eb1589
paper_authors.name: Simon Dixon
paper_authors._id: 66cd4bff2299fbc852eb158a
paper_authors.name: Wenhu Chen
paper_authors._id: 66cd4bff2299fbc852eb158b
paper_authors.name: Wehhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:25:47.473Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 66cd4bff2299fbc852eb158c
paper_authors.name: Xingjian Du
paper_authors._id: 66cd4bff2299fbc852eb158d
paper_authors.name: Xingwei Qu
paper_authors._id: 66cd4bff2299fbc852eb158e
paper_authors.name: Xu Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T13:47:09.935Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 66cd4bff2299fbc852eb158f
paper_authors.name: Yizhi Li
paper_authors._id: 66cd4bff2299fbc852eb1590
paper_authors.name: Zeyue Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:23:54.181Z
paper_authors._id: 6504409b52ca06fef957266d
paper_authors.avatarUrl: /avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
paper_authors.fullname: Zeyue
paper_authors.type: user
paper_authors.user: Zeyue7
paper_authors._id: 66cd4bff2299fbc852eb1591
paper_authors.name: Zhiyong Wu
paper_authors._id: 66cd4bff2299fbc852eb1592
paper_authors.name: Zhizheng Wu
paper_authors._id: 66cd4bff2299fbc852eb1593
paper_authors.name: Ziyang Ma
paper_authors._id: 66cd4bff2299fbc852eb1594
paper_authors.name: Ziyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T10:32:56.237Z
paper_authors._id: 666e9aef15b60b3fba91fce1
paper_authors.avatarUrl: /avatars/2b27cc8895e6f21c02b751791a568962.svg
paper_authors.fullname: Ziyu Wang
paper_authors.type: user
paper_authors.user: ZzWaang
paper_title: Foundation Models for Music: A Survey
paper_summary: In recent years, foundation models (FMs) such as large language models (LLMs)
  and latent diffusion models (LDMs) have profoundly impacted diverse sectors,
  including music. This comprehensive review examines state-of-the-art (SOTA)
  pre-trained models and foundation models in music, spanning from representation
  learning, generative learning and multimodal learning. We first contextualise
  the significance of music in various industries and trace the evolution of AI
  in music. By delineating the modalities targeted by foundation models, we
  discover many of the music representations are underexplored in FM development.
  Then, emphasis is placed on the lack of versatility of previous methods on
  diverse music applications, along with the potential of FMs in music
  understanding, generation and medical application. By comprehensively exploring
  the details of the model pre-training paradigm, architectural choices,
  tokenisation, finetuning methodologies and controllability, we emphasise the
  important topics that should have been well explored, like instruction tuning
  and in-context learning, scaling law and emergent ability, as well as
  long-sequence modelling etc. A dedicated section presents insights into music
  agents, accompanied by a thorough analysis of datasets and evaluations
  essential for pre-training and downstream tasks. Finally, by underscoring the
  vital importance of ethical considerations, we advocate that following research
  on FM for music should focus more on such issues as interpretability,
  transparency, human responsibility, and copyright issues. The paper offers
  insights into future challenges and trends on FMs for music, aiming to shape
  the trajectory of human-AI collaboration in the music realm.
paper_ai_summary: A review of foundation models in music, including large language models and latent diffusion models, highlights their impact, architectural choices, and the need for ethical considerations in music applications.
paper_ai_keywords: large language models latent diffusion models representation learning generative learning multimodal learning instruction tuning in-context learning scaling law emergent ability long-sequence modelling music agents