MERT: Acoustic Music Understanding Model with Large-Scale
  Self-supervised Training
Self-supervised learning (SSL) has recently emerged as a promising paradigm
for training generalisable models on large-scale data in the fields of vision,
text, and speech. Although SSL has been proven effective in speech and audio,
its application to music audio has yet to be thoroughly explored. This is
primarily due to the distinctive challenges associated with modelling musical
knowledge, particularly its tonal and pitched characteristics of music. To
address this research gap, we propose an acoustic Music undERstanding model
with large-scale self-supervised Training (MERT), which incorporates teacher
models to provide pseudo labels in the masked language modelling (MLM) style
acoustic pre-training. In our exploration, we identified a superior combination
of teacher models, which outperforms conventional speech and audio approaches
in terms of performance. This combination includes an acoustic teacher based on
Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical
teacher based on the Constant-Q Transform (CQT). These teachers effectively
guide our student model, a BERT-style transformer encoder, to better model
music audio. In addition, we introduce an in-batch noise mixture augmentation
to enhance the representation robustness. Furthermore, we explore a wide range
of settings to overcome the instability in acoustic language model
pre-training, which allows our designed paradigm to scale from 95M to 330M
parameters. Experimental results indicate that our model can generalise and
perform well on 14 music understanding tasks and attains state-of-the-art
(SOTA) overall scores. The code and models are online:
https://github.com/yizhilll/MERT.
64795c84582c88b4812d1fea
Yizhi Li
claimed_verified
2024-05-31T06:55:14.094Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
64795c84582c88b4812d1feb
Ruibin Yuan
admin_assigned
2023-06-07T13:51:15.174Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
64795c84582c88b4812d1fec
Ge Zhang
extracted_confirmed
2023-06-02T03:25:16.897Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
64795c84582c88b4812d1fed
Yinghao Ma
admin_assigned
2023-06-07T13:50:54.905Z
6410665d5364a661bee22524
/avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
Yinghao Ma
user
nicolaus625
64795c84582c88b4812d1fee
Xingran Chen
admin_assigned
2023-06-07T13:51:05.561Z
61c30f612250077bffed7636
https://cdn-avatars.huggingface.co/v1/production/uploads/1647200227141-61c30f612250077bffed7636.jpeg
Xingran Chen
user
chenxran
64795c84582c88b4812d1fef
Hanzhi Yin
admin_assigned
2023-06-07T13:51:36.645Z
637eea070bb954fb10e37435
/avatars/039005b11450f154b4416f1d207073bd.svg
Hanzhi Yin
user
int0thewind
64795c84582c88b4812d1ff0
Chenghua Lin
admin_assigned
2023-06-07T14:10:46.268Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
64795c84582c88b4812d1ff1
Anton Ragni
64795c84582c88b4812d1ff2
Emmanouil Benetos
claimed_verified
2024-08-05T10:24:19.119Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
64795c84582c88b4812d1ff3
Norbert Gyenge
64795c84582c88b4812d1ff4
Roger Dannenberg
64795c84582c88b4812d1ff5
Ruibo Liu
admin_assigned
2023-06-07T14:11:55.740Z
5e91f1554957053f606489ff
https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
Ruibo Liu
user
agi-css
64795c84582c88b4812d1ff6
Wenhu Chen
64795c84582c88b4812d1ff7
Gus Xia
64795c84582c88b4812d1ff8
Yemin Shi
64795c84582c88b4812d1ff9
Wenhao Huang
claimed_verified
2024-10-13T20:23:45.966Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
64795c84582c88b4812d1ffa
Yike Guo
admin_assigned
2023-06-07T13:48:20.569Z
64762b325f70f9b2d0ade28e
/avatars/1f37382724b475ece805f943a8858acd.svg
Yike Guo
user
SuaLily
64795c84582c88b4812d1ffb
Jie Fu
claimed_verified
2023-06-02T07:16:47.479Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
MERT: Acoustic Music Understanding Model with Large-Scale
  Self-supervised Training
Self-supervised learning (SSL) has recently emerged as a promising paradigm
for training generalisable models on large-scale data in the fields of vision,
text, and speech. Although SSL has been proven effective in speech and audio,
its application to music audio has yet to be thoroughly explored. This is
primarily due to the distinctive challenges associated with modelling musical
knowledge, particularly its tonal and pitched characteristics of music. To
address this research gap, we propose an acoustic Music undERstanding model
with large-scale self-supervised Training (MERT), which incorporates teacher
models to provide pseudo labels in the masked language modelling (MLM) style
acoustic pre-training. In our exploration, we identified a superior combination
of teacher models, which outperforms conventional speech and audio approaches
in terms of performance. This combination includes an acoustic teacher based on
Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical
teacher based on the Constant-Q Transform (CQT). These teachers effectively
guide our student model, a BERT-style transformer encoder, to better model
music audio. In addition, we introduce an in-batch noise mixture augmentation
to enhance the representation robustness. Furthermore, we explore a wide range
of settings to overcome the instability in acoustic language model
pre-training, which allows our designed paradigm to scale from 95M to 330M
parameters. Experimental results indicate that our model can generalise and
perform well on 14 music understanding tasks and attains state-of-the-art
(SOTA) overall scores. The code and models are online:
https://github.com/yizhilll/MERT.
MERT, a self-supervised learning model for music audio, uses a combination of RVQ-VAE and CQT-based teacher models to achieve state-of-the-art performance across 14 music understanding tasks.
self-supervised learning MERT masked language modelling RVQ-VAE Constant-Q Transform BERT-style transformer encoder in-batch noise mixture augmentation acoustic language model pre-training