title: MERT: Acoustic Music Understanding Model with Large-Scale
    Self-supervised Training
summary: Self-supervised learning (SSL) has recently emerged as a promising paradigm
  for training generalisable models on large-scale data in the fields of vision,
  text, and speech. Although SSL has been proven effective in speech and audio,
  its application to music audio has yet to be thoroughly explored. This is
  primarily due to the distinctive challenges associated with modelling musical
  knowledge, particularly its tonal and pitched characteristics of music. To
  address this research gap, we propose an acoustic Music undERstanding model
  with large-scale self-supervised Training (MERT), which incorporates teacher
  models to provide pseudo labels in the masked language modelling (MLM) style
  acoustic pre-training. In our exploration, we identified a superior combination
  of teacher models, which outperforms conventional speech and audio approaches
  in terms of performance. This combination includes an acoustic teacher based on
  Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical
  teacher based on the Constant-Q Transform (CQT). These teachers effectively
  guide our student model, a BERT-style transformer encoder, to better model
  music audio. In addition, we introduce an in-batch noise mixture augmentation
  to enhance the representation robustness. Furthermore, we explore a wide range
  of settings to overcome the instability in acoustic language model
  pre-training, which allows our designed paradigm to scale from 95M to 330M
  parameters. Experimental results indicate that our model can generalise and
  perform well on 14 music understanding tasks and attains state-of-the-art
  (SOTA) overall scores. The code and models are online:
  https://github.com/yizhilll/MERT.
paper_authors._id: 64795c84582c88b4812d1fea
paper_authors.name: Yizhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:55:14.094Z
paper_authors._id: 6382252f54421460665ec501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
paper_authors.fullname: Yizhi Li
paper_authors.type: user
paper_authors.user: yizhilll
paper_authors._id: 64795c84582c88b4812d1feb
paper_authors.name: Ruibin Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T13:51:15.174Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 64795c84582c88b4812d1fec
paper_authors.name: Ge Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-02T03:25:16.897Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 64795c84582c88b4812d1fed
paper_authors.name: Yinghao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T13:50:54.905Z
paper_authors._id: 6410665d5364a661bee22524
paper_authors.avatarUrl: /avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
paper_authors.fullname: Yinghao Ma
paper_authors.type: user
paper_authors.user: nicolaus625
paper_authors._id: 64795c84582c88b4812d1fee
paper_authors.name: Xingran Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T13:51:05.561Z
paper_authors._id: 61c30f612250077bffed7636
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647200227141-61c30f612250077bffed7636.jpeg
paper_authors.fullname: Xingran Chen
paper_authors.type: user
paper_authors.user: chenxran
paper_authors._id: 64795c84582c88b4812d1fef
paper_authors.name: Hanzhi Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T13:51:36.645Z
paper_authors._id: 637eea070bb954fb10e37435
paper_authors.avatarUrl: /avatars/039005b11450f154b4416f1d207073bd.svg
paper_authors.fullname: Hanzhi Yin
paper_authors.type: user
paper_authors.user: int0thewind
paper_authors._id: 64795c84582c88b4812d1ff0
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T14:10:46.268Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_authors._id: 64795c84582c88b4812d1ff1
paper_authors.name: Anton Ragni
paper_authors._id: 64795c84582c88b4812d1ff2
paper_authors.name: Emmanouil Benetos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T10:24:19.119Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 64795c84582c88b4812d1ff3
paper_authors.name: Norbert Gyenge
paper_authors._id: 64795c84582c88b4812d1ff4
paper_authors.name: Roger Dannenberg
paper_authors._id: 64795c84582c88b4812d1ff5
paper_authors.name: Ruibo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T14:11:55.740Z
paper_authors._id: 5e91f1554957053f606489ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
paper_authors.fullname: Ruibo Liu
paper_authors.type: user
paper_authors.user: agi-css
paper_authors._id: 64795c84582c88b4812d1ff6
paper_authors.name: Wenhu Chen
paper_authors._id: 64795c84582c88b4812d1ff7
paper_authors.name: Gus Xia
paper_authors._id: 64795c84582c88b4812d1ff8
paper_authors.name: Yemin Shi
paper_authors._id: 64795c84582c88b4812d1ff9
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:23:45.966Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 64795c84582c88b4812d1ffa
paper_authors.name: Yike Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T13:48:20.569Z
paper_authors._id: 64762b325f70f9b2d0ade28e
paper_authors.avatarUrl: /avatars/1f37382724b475ece805f943a8858acd.svg
paper_authors.fullname: Yike Guo
paper_authors.type: user
paper_authors.user: SuaLily
paper_authors._id: 64795c84582c88b4812d1ffb
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T07:16:47.479Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: MERT: Acoustic Music Understanding Model with Large-Scale
    Self-supervised Training
paper_summary: Self-supervised learning (SSL) has recently emerged as a promising paradigm
  for training generalisable models on large-scale data in the fields of vision,
  text, and speech. Although SSL has been proven effective in speech and audio,
  its application to music audio has yet to be thoroughly explored. This is
  primarily due to the distinctive challenges associated with modelling musical
  knowledge, particularly its tonal and pitched characteristics of music. To
  address this research gap, we propose an acoustic Music undERstanding model
  with large-scale self-supervised Training (MERT), which incorporates teacher
  models to provide pseudo labels in the masked language modelling (MLM) style
  acoustic pre-training. In our exploration, we identified a superior combination
  of teacher models, which outperforms conventional speech and audio approaches
  in terms of performance. This combination includes an acoustic teacher based on
  Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical
  teacher based on the Constant-Q Transform (CQT). These teachers effectively
  guide our student model, a BERT-style transformer encoder, to better model
  music audio. In addition, we introduce an in-batch noise mixture augmentation
  to enhance the representation robustness. Furthermore, we explore a wide range
  of settings to overcome the instability in acoustic language model
  pre-training, which allows our designed paradigm to scale from 95M to 330M
  parameters. Experimental results indicate that our model can generalise and
  perform well on 14 music understanding tasks and attains state-of-the-art
  (SOTA) overall scores. The code and models are online:
  https://github.com/yizhilll/MERT.
paper_ai_summary: MERT, a self-supervised learning model for music audio, uses a combination of RVQ-VAE and CQT-based teacher models to achieve state-of-the-art performance across 14 music understanding tasks.
paper_ai_keywords: self-supervised learning MERT masked language modelling RVQ-VAE Constant-Q Transform BERT-style transformer encoder in-batch noise mixture augmentation acoustic language model pre-training