title: ChatMusician: Understanding and Generating Music Intrinsically with LLM
summary: While Large Language Models (LLMs) demonstrate impressive capabilities in
  text generation, we find that their ability has yet to be generalized to music,
  humanity's creative language. We introduce ChatMusician, an open-source LLM
  that integrates intrinsic musical abilities. It is based on continual
  pre-training and finetuning LLaMA2 on a text-compatible music representation,
  ABC notation, and the music is treated as a second language. ChatMusician can
  understand and generate music with a pure text tokenizer without any external
  multi-modal neural structures or tokenizers. Interestingly, endowing musical
  abilities does not harm language abilities, even achieving a slightly higher
  MMLU score. Our model is capable of composing well-structured, full-length
  music, conditioned on texts, chords, melodies, motifs, musical forms, etc,
  surpassing GPT-4 baseline. On our meticulously curated college-level music
  understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and
  GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs
  can be an excellent compressor for music, but there remains significant
  territory to be conquered. We release our 4B token music-language corpora
  MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.
paper_authors._id: 65dd6a140481e2a395d53127
paper_authors.name: Ruibin Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:03:36.467Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 65dd6a140481e2a395d53128
paper_authors.name: Hanfeng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:03:52.608Z
paper_authors._id: 638df7f3d975533b16a5a739
paper_authors.avatarUrl: /avatars/dbc8388e7c32c437b66619ea9b3ad827.svg
paper_authors.fullname: Hanfeng Lin
paper_authors.type: user
paper_authors.user: agent404
paper_authors._id: 65dd6a140481e2a395d53129
paper_authors.name: Yi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T08:12:03.487Z
paper_authors._id: 649f3875c6c143d7c05443af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f3875c6c143d7c05443af/MJjMZvxs8OoQlSSsdLTZE.jpeg
paper_authors.fullname: Yi
paper_authors.type: user
paper_authors.user: EZMONYI
paper_authors._id: 65dd6a140481e2a395d5312a
paper_authors.name: Zeyue Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:25:09.693Z
paper_authors._id: 6504409b52ca06fef957266d
paper_authors.avatarUrl: /avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
paper_authors.fullname: Zeyue
paper_authors.type: user
paper_authors.user: Zeyue7
paper_authors._id: 65dd6a140481e2a395d5312b
paper_authors.name: Shangda Wu
paper_authors._id: 65dd6a140481e2a395d5312c
paper_authors.name: Tianhao Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T09:04:20.939Z
paper_authors._id: 61976063b1099a2392101960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61976063b1099a2392101960/DRlAoI6cD6yyomQfWRTU5.png
paper_authors.fullname: Tianhao Shen
paper_authors.type: user
paper_authors.user: magnetic
paper_authors._id: 65dd6a140481e2a395d5312d
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T08:12:07.364Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 65dd6a140481e2a395d5312e
paper_authors.name: Yuhang Wu
paper_authors._id: 65dd6a140481e2a395d5312f
paper_authors.name: Cong Liu
paper_authors._id: 65dd6a140481e2a395d53130
paper_authors.name: Ziya Zhou
paper_authors._id: 65dd6a140481e2a395d53131
paper_authors.name: Ziyang Ma
paper_authors._id: 65dd6a140481e2a395d53132
paper_authors.name: Liumeng Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:06:27.079Z
paper_authors._id: 6290e961473e457463a53248
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6290e961473e457463a53248/-58Dp5uHvdjs9yOupAMs0.jpeg
paper_authors.fullname: Liumeng Xue
paper_authors.type: user
paper_authors.user: lmxue
paper_authors._id: 65dd6a140481e2a395d53133
paper_authors.name: Ziyu Wang
paper_authors._id: 65dd6a140481e2a395d53134
paper_authors.name: Qin Liu
paper_authors._id: 65dd6a140481e2a395d53135
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T08:12:05.684Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 65dd6a140481e2a395d53136
paper_authors.name: Yizhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:53:58.640Z
paper_authors._id: 6382252f54421460665ec501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
paper_authors.fullname: Yizhi Li
paper_authors.type: user
paper_authors.user: yizhilll
paper_authors._id: 65dd6a140481e2a395d53137
paper_authors.name: Yinghao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:12:31.637Z
paper_authors._id: 6410665d5364a661bee22524
paper_authors.avatarUrl: /avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
paper_authors.fullname: Yinghao Ma
paper_authors.type: user
paper_authors.user: nicolaus625
paper_authors._id: 65dd6a140481e2a395d53138
paper_authors.name: Yiming Liang
paper_authors._id: 65dd6a140481e2a395d53139
paper_authors.name: Xiaowei Chi
paper_authors._id: 65dd6a140481e2a395d5313a
paper_authors.name: Ruibo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:13:28.284Z
paper_authors._id: 5e91f1554957053f606489ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
paper_authors.fullname: Ruibo Liu
paper_authors.type: user
paper_authors.user: agi-css
paper_authors._id: 65dd6a140481e2a395d5313b
paper_authors.name: Zili Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:13:36.104Z
paper_authors._id: 6410998db27543634e3aacc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: MarkWang
paper_authors._id: 65dd6a140481e2a395d5313c
paper_authors.name: Pengfei Li
paper_authors._id: 65dd6a140481e2a395d5313d
paper_authors.name: Jingcheng Wu
paper_authors._id: 65dd6a140481e2a395d5313e
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:15:44.000Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_authors._id: 65dd6a140481e2a395d5313f
paper_authors.name: Qifeng Liu
paper_authors._id: 65dd6a140481e2a395d53140
paper_authors.name: Tao Jiang
paper_authors._id: 65dd6a140481e2a395d53141
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:31:44.605Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 65dd6a140481e2a395d53142
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:21:59.833Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 65dd6a140481e2a395d53143
paper_authors.name: Emmanouil Benetos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T10:24:13.423Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 65dd6a140481e2a395d53144
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T09:04:18.809Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 65dd6a140481e2a395d53145
paper_authors.name: Gus Xia
paper_authors._id: 65dd6a140481e2a395d53146
paper_authors.name: Roger Dannenberg
paper_authors._id: 65dd6a140481e2a395d53147
paper_authors.name: Wei Xue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:12:45.191Z
paper_authors._id: 6628adb14277eae0da5eee28
paper_authors.avatarUrl: /avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
paper_authors.fullname: HKUST Audio
paper_authors.type: user
paper_authors.user: HKUST-Audio
paper_authors._id: 65dd6a140481e2a395d53148
paper_authors.name: Shiyin Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:23:04.207Z
paper_authors._id: 65dda40f22d4f4074585ecc6
paper_authors.avatarUrl: /avatars/b616537ad7664d6f0d24c2b6ff098098.svg
paper_authors.fullname: Shiyin Kang
paper_authors.type: user
paper_authors.user: kangshiyin
paper_authors._id: 65dd6a140481e2a395d53149
paper_authors.name: Yike Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:22:56.294Z
paper_authors._id: 64762b325f70f9b2d0ade28e
paper_authors.avatarUrl: /avatars/1f37382724b475ece805f943a8858acd.svg
paper_authors.fullname: Yike Guo
paper_authors.type: user
paper_authors.user: SuaLily
paper_title: ChatMusician: Understanding and Generating Music Intrinsically with LLM
paper_summary: While Large Language Models (LLMs) demonstrate impressive capabilities in
  text generation, we find that their ability has yet to be generalized to music,
  humanity's creative language. We introduce ChatMusician, an open-source LLM
  that integrates intrinsic musical abilities. It is based on continual
  pre-training and finetuning LLaMA2 on a text-compatible music representation,
  ABC notation, and the music is treated as a second language. ChatMusician can
  understand and generate music with a pure text tokenizer without any external
  multi-modal neural structures or tokenizers. Interestingly, endowing musical
  abilities does not harm language abilities, even achieving a slightly higher
  MMLU score. Our model is capable of composing well-structured, full-length
  music, conditioned on texts, chords, melodies, motifs, musical forms, etc,
  surpassing GPT-4 baseline. On our meticulously curated college-level music
  understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and
  GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs
  can be an excellent compressor for music, but there remains significant
  territory to be conquered. We release our 4B token music-language corpora
  MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.
paper_ai_summary: ChatMusician, an open-source LLM with intrinsic musical abilities, outperforms GPT-4 in music generation and understanding tasks using text-compatible music notation.
paper_ai_keywords: LLMs ChatMusician continual pre-training finetuning LLaMA2 ABC notation text tokenizer MMLU score MusicTheoryBench MusicPile