title: MuPT: A Generative Symbolic Music Pretrained Transformer
summary: In this paper, we explore the application of Large Language Models (LLMs) to
  the pre-training of music. While the prevalent use of MIDI in music modeling is
  well-established, our findings suggest that LLMs are inherently more compatible
  with ABC Notation, which aligns more closely with their design and strengths,
  thereby enhancing the model's performance in musical composition. To address
  the challenges associated with misaligned measures from different tracks during
  generation, we propose the development of a Synchronized
  Multi-Track ABC Notation (SMT-ABC Notation),
  which aims to preserve coherence across multiple musical tracks. Our
  contributions include a series of models capable of handling up to 8192 tokens,
  covering 90\% of the symbolic music data in our training set. Furthermore, we
  explore the implications of the Symbolic Music
  Scaling Law (SMS Law) on model performance. The results
  indicate a promising direction for future research in music generation,
  offering extensive resources for community-led research through our open-source
  contributions.
paper_authors._id: 6615ef8c45336ca7744f3583
paper_authors.name: Xingwei Qu
paper_authors._id: 6615ef8c45336ca7744f3584
paper_authors.name: Yuelin Bai
paper_authors._id: 6615ef8c45336ca7744f3585
paper_authors.name: Yinghao Ma
paper_authors._id: 6615ef8c45336ca7744f3586
paper_authors.name: Ziya Zhou
paper_authors._id: 6615ef8c45336ca7744f3587
paper_authors.name: Ka Man Lo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:47:41.055Z
paper_authors._id: 649ad817c5d076065a514a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649ad817c5d076065a514a41/WsGymqaiZFsx3jDtMCycl.jpeg
paper_authors.fullname: Ka Man Lo
paper_authors.type: user
paper_authors.user: kamanphoebe
paper_authors._id: 6615ef8c45336ca7744f3588
paper_authors.name: Jiaheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:54.527Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 6615ef8c45336ca7744f3589
paper_authors.name: Ruibin Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:57:32.770Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 6615ef8c45336ca7744f358a
paper_authors.name: Lejun Min
paper_authors._id: 6615ef8c45336ca7744f358b
paper_authors.name: Xueling Liu
paper_authors._id: 6615ef8c45336ca7744f358c
paper_authors.name: Tianyu Zhang
paper_authors._id: 6615ef8c45336ca7744f358d
paper_authors.name: Xinrun Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:53:34.584Z
paper_authors._id: 654907a4a1faff97850c4eff
paper_authors.avatarUrl: /avatars/458c90151614bc7f116943b6e67d6b8a.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: dododododo
paper_authors._id: 6615ef8c45336ca7744f358e
paper_authors.name: Shuyue Guo
paper_authors._id: 6615ef8c45336ca7744f358f
paper_authors.name: Yiming Liang
paper_authors._id: 6615ef8c45336ca7744f3590
paper_authors.name: Yizhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:53:36.464Z
paper_authors._id: 6382252f54421460665ec501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
paper_authors.fullname: Yizhi Li
paper_authors.type: user
paper_authors.user: yizhilll
paper_authors._id: 6615ef8c45336ca7744f3591
paper_authors.name: Shangda Wu
paper_authors._id: 6615ef8c45336ca7744f3592
paper_authors.name: Junting Zhou
paper_authors._id: 6615ef8c45336ca7744f3593
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:57:49.141Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 6615ef8c45336ca7744f3594
paper_authors.name: Ziyang Ma
paper_authors._id: 6615ef8c45336ca7744f3595
paper_authors.name: Fengze Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:58:00.703Z
paper_authors._id: 65d22ca28842349c64e3e7f1
paper_authors.avatarUrl: /avatars/38720c9fa9dbf9cf1a73cc4d418836e1.svg
paper_authors.fullname: Fengze Han
paper_authors.type: user
paper_authors.user: Fengze
paper_authors._id: 6615ef8c45336ca7744f3596
paper_authors.name: Wei Xue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:12:13.207Z
paper_authors._id: 6628adb14277eae0da5eee28
paper_authors.avatarUrl: /avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
paper_authors.fullname: HKUST Audio
paper_authors.type: user
paper_authors.user: HKUST-Audio
paper_authors._id: 6615ef8c45336ca7744f3597
paper_authors.name: Gus Xia
paper_authors._id: 6615ef8c45336ca7744f3598
paper_authors.name: Emmanouil Benetos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T10:24:11.931Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 6615ef8c45336ca7744f3599
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:21:02.656Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 6615ef8c45336ca7744f359a
paper_authors.name: Chenghua Lin
paper_authors._id: 6615ef8c45336ca7744f359b
paper_authors.name: Xu Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T13:47:24.072Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 6615ef8c45336ca7744f359c
paper_authors.name: Stephen W. Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:26:00.069Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 6615ef8c45336ca7744f359d
paper_authors.name: Wenhu Chen
paper_authors._id: 6615ef8c45336ca7744f359e
paper_authors.name: Jie Fu
paper_authors._id: 6615ef8c45336ca7744f359f
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:53:32.871Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_title: MuPT: A Generative Symbolic Music Pretrained Transformer
paper_summary: In this paper, we explore the application of Large Language Models (LLMs) to
  the pre-training of music. While the prevalent use of MIDI in music modeling is
  well-established, our findings suggest that LLMs are inherently more compatible
  with ABC Notation, which aligns more closely with their design and strengths,
  thereby enhancing the model's performance in musical composition. To address
  the challenges associated with misaligned measures from different tracks during
  generation, we propose the development of a Synchronized
  Multi-Track ABC Notation (SMT-ABC Notation),
  which aims to preserve coherence across multiple musical tracks. Our
  contributions include a series of models capable of handling up to 8192 tokens,
  covering 90\% of the symbolic music data in our training set. Furthermore, we
  explore the implications of the Symbolic Music
  Scaling Law (SMS Law) on model performance. The results
  indicate a promising direction for future research in music generation,
  offering extensive resources for community-led research through our open-source
  contributions.
paper_ai_summary: Large Language Models are adapted for music pre-training using ABC Notation instead of MIDI, leading to enhanced musical composition through synchronized multi-track handling and exploration of the Symbolic Music Scaling Law.
paper_ai_keywords: Large Language Models LLMs MIDI ABC Notation SMT-ABC Notation symbolic music Symbolic Music Scaling Law SMS Law