ChatMusician: Understanding and Generating Music Intrinsically with LLM
While Large Language Models (LLMs) demonstrate impressive capabilities in
text generation, we find that their ability has yet to be generalized to music,
humanity's creative language. We introduce ChatMusician, an open-source LLM
that integrates intrinsic musical abilities. It is based on continual
pre-training and finetuning LLaMA2 on a text-compatible music representation,
ABC notation, and the music is treated as a second language. ChatMusician can
understand and generate music with a pure text tokenizer without any external
multi-modal neural structures or tokenizers. Interestingly, endowing musical
abilities does not harm language abilities, even achieving a slightly higher
MMLU score. Our model is capable of composing well-structured, full-length
music, conditioned on texts, chords, melodies, motifs, musical forms, etc,
surpassing GPT-4 baseline. On our meticulously curated college-level music
understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and
GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs
can be an excellent compressor for music, but there remains significant
territory to be conquered. We release our 4B token music-language corpora
MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.
65dd6a140481e2a395d53127
Ruibin Yuan
admin_assigned
2024-02-27T09:03:36.467Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
65dd6a140481e2a395d53128
Hanfeng Lin
admin_assigned
2024-02-27T09:03:52.608Z
638df7f3d975533b16a5a739
/avatars/dbc8388e7c32c437b66619ea9b3ad827.svg
Hanfeng Lin
user
agent404
65dd6a140481e2a395d53129
Yi Wang
claimed_verified
2024-02-27T08:12:03.487Z
649f3875c6c143d7c05443af
https://cdn-avatars.huggingface.co/v1/production/uploads/649f3875c6c143d7c05443af/MJjMZvxs8OoQlSSsdLTZE.jpeg
Yi
user
EZMONYI
65dd6a140481e2a395d5312a
Zeyue Tian
claimed_verified
2025-03-16T21:25:09.693Z
6504409b52ca06fef957266d
/avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
Zeyue
user
Zeyue7
65dd6a140481e2a395d5312b
Shangda Wu
65dd6a140481e2a395d5312c
Tianhao Shen
claimed_verified
2024-02-27T09:04:20.939Z
61976063b1099a2392101960
https://cdn-avatars.huggingface.co/v1/production/uploads/61976063b1099a2392101960/DRlAoI6cD6yyomQfWRTU5.png
Tianhao Shen
user
magnetic
65dd6a140481e2a395d5312d
Ge Zhang
claimed_verified
2024-02-27T08:12:07.364Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
65dd6a140481e2a395d5312e
Yuhang Wu
65dd6a140481e2a395d5312f
Cong Liu
65dd6a140481e2a395d53130
Ziya Zhou
65dd6a140481e2a395d53131
Ziyang Ma
65dd6a140481e2a395d53132
Liumeng Xue
admin_assigned
2024-02-27T09:06:27.079Z
6290e961473e457463a53248
https://cdn-avatars.huggingface.co/v1/production/uploads/6290e961473e457463a53248/-58Dp5uHvdjs9yOupAMs0.jpeg
Liumeng Xue
user
lmxue
65dd6a140481e2a395d53133
Ziyu Wang
65dd6a140481e2a395d53134
Qin Liu
65dd6a140481e2a395d53135
Tianyu Zheng
claimed_verified
2024-02-27T08:12:05.684Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
65dd6a140481e2a395d53136
Yizhi Li
claimed_verified
2024-05-31T06:53:58.640Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
65dd6a140481e2a395d53137
Yinghao Ma
admin_assigned
2024-02-27T09:12:31.637Z
6410665d5364a661bee22524
/avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
Yinghao Ma
user
nicolaus625
65dd6a140481e2a395d53138
Yiming Liang
65dd6a140481e2a395d53139
Xiaowei Chi
65dd6a140481e2a395d5313a
Ruibo Liu
admin_assigned
2024-02-27T09:13:28.284Z
5e91f1554957053f606489ff
https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
Ruibo Liu
user
agi-css
65dd6a140481e2a395d5313b
Zili Wang
admin_assigned
2024-02-27T09:13:36.104Z
6410998db27543634e3aacc3
https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
Zili Wang
user
MarkWang
65dd6a140481e2a395d5313c
Pengfei Li
65dd6a140481e2a395d5313d
Jingcheng Wu
65dd6a140481e2a395d5313e
Chenghua Lin
admin_assigned
2024-02-27T09:15:44.000Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
65dd6a140481e2a395d5313f
Qifeng Liu
65dd6a140481e2a395d53140
Tao Jiang
65dd6a140481e2a395d53141
Wenhao Huang
claimed_verified
2024-10-13T20:31:44.605Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
65dd6a140481e2a395d53142
Wenhu Chen
admin_assigned
2024-02-27T09:21:59.833Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
65dd6a140481e2a395d53143
Emmanouil Benetos
claimed_verified
2024-08-05T10:24:13.423Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
65dd6a140481e2a395d53144
Jie Fu
claimed_verified
2024-02-27T09:04:18.809Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
65dd6a140481e2a395d53145
Gus Xia
65dd6a140481e2a395d53146
Roger Dannenberg
65dd6a140481e2a395d53147
Wei Xue
claimed_verified
2024-04-24T11:12:45.191Z
6628adb14277eae0da5eee28
/avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
HKUST Audio
user
HKUST-Audio
65dd6a140481e2a395d53148
Shiyin Kang
admin_assigned
2024-02-27T09:23:04.207Z
65dda40f22d4f4074585ecc6
/avatars/b616537ad7664d6f0d24c2b6ff098098.svg
Shiyin Kang
user
kangshiyin
65dd6a140481e2a395d53149
Yike Guo
admin_assigned
2024-02-27T09:22:56.294Z
64762b325f70f9b2d0ade28e
/avatars/1f37382724b475ece805f943a8858acd.svg
Yike Guo
user
SuaLily
ChatMusician: Understanding and Generating Music Intrinsically with LLM
While Large Language Models (LLMs) demonstrate impressive capabilities in
text generation, we find that their ability has yet to be generalized to music,
humanity's creative language. We introduce ChatMusician, an open-source LLM
that integrates intrinsic musical abilities. It is based on continual
pre-training and finetuning LLaMA2 on a text-compatible music representation,
ABC notation, and the music is treated as a second language. ChatMusician can
understand and generate music with a pure text tokenizer without any external
multi-modal neural structures or tokenizers. Interestingly, endowing musical
abilities does not harm language abilities, even achieving a slightly higher
MMLU score. Our model is capable of composing well-structured, full-length
music, conditioned on texts, chords, melodies, motifs, musical forms, etc,
surpassing GPT-4 baseline. On our meticulously curated college-level music
understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and
GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs
can be an excellent compressor for music, but there remains significant
territory to be conquered. We release our 4B token music-language corpora
MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.
ChatMusician, an open-source LLM with intrinsic musical abilities, outperforms GPT-4 in music generation and understanding tasks using text-compatible music notation.
LLMs ChatMusician continual pre-training finetuning LLaMA2 ABC notation text tokenizer MMLU score MusicTheoryBench MusicPile