CultureMERT: Continual Pre-Training for Cross-Cultural Music
  Representation Learning
Recent advances in music foundation models have improved audio representation
learning, yet their effectiveness across diverse musical traditions remains
limited. We introduce CultureMERT-95M, a multi-culturally adapted foundation
model developed to enhance cross-cultural music representation learning and
understanding. To achieve this, we propose a two-stage continual pre-training
strategy that integrates learning rate re-warming and re-decaying, enabling
stable adaptation even with limited computational resources. Training on a
650-hour multi-cultural data mix, comprising Greek, Turkish, and Indian music
traditions, results in an average improvement of 4.9% in ROC-AUC and AP across
diverse non-Western music auto-tagging tasks, surpassing prior
state-of-the-art, with minimal forgetting on Western-centric benchmarks. We
further investigate task arithmetic, an alternative approach to multi-cultural
adaptation that merges single-culture adapted models in the weight space. Task
arithmetic performs on par with our multi-culturally trained model on
non-Western auto-tagging tasks and shows no regression on Western datasets.
Cross-cultural evaluation reveals that single-culture models transfer with
varying effectiveness across musical traditions, whereas the multi-culturally
adapted model achieves the best overall performance. To support research on
world music representation learning, we publicly release CultureMERT-95M and
CultureMERT-TA-95M, fostering the development of more culturally aware music
foundation models.
685a887b0e4ad7e219758725
Angelos-Nikolaos Kanatas
claimed_verified
2025-06-24T12:43:08.653Z
640e517d612e7f36f90f0016
/avatars/48104e174fe028aef246e889e468f940.svg
Angelos-Nikolaos Kanatas
user
akanatas
685a887b0e4ad7e219758726
Charilaos Papaioannou
claimed_verified
2025-06-24T16:25:12.261Z
65c0ae0cfdb8d33cef6947cf
/avatars/60cdbf1fc85d433d0fb4ff4a240f39cd.svg
Charis Papaioannou
user
pxaris
685a887b0e4ad7e219758727
Alexandros Potamianos
claimed_verified
2025-06-25T08:09:15.338Z
63bfaa9887619d1458d24f8a
/avatars/17fcd55e19103eba3f19a82371c00c28.svg
Alexandros Potamianos
user
apotam
CultureMERT: Continual Pre-Training for Cross-Cultural Music
  Representation Learning
Recent advances in music foundation models have improved audio representation
learning, yet their effectiveness across diverse musical traditions remains
limited. We introduce CultureMERT-95M, a multi-culturally adapted foundation
model developed to enhance cross-cultural music representation learning and
understanding. To achieve this, we propose a two-stage continual pre-training
strategy that integrates learning rate re-warming and re-decaying, enabling
stable adaptation even with limited computational resources. Training on a
650-hour multi-cultural data mix, comprising Greek, Turkish, and Indian music
traditions, results in an average improvement of 4.9% in ROC-AUC and AP across
diverse non-Western music auto-tagging tasks, surpassing prior
state-of-the-art, with minimal forgetting on Western-centric benchmarks. We
further investigate task arithmetic, an alternative approach to multi-cultural
adaptation that merges single-culture adapted models in the weight space. Task
arithmetic performs on par with our multi-culturally trained model on
non-Western auto-tagging tasks and shows no regression on Western datasets.
Cross-cultural evaluation reveals that single-culture models transfer with
varying effectiveness across musical traditions, whereas the multi-culturally
adapted model achieves the best overall performance. To support research on
world music representation learning, we publicly release CultureMERT-95M and
CultureMERT-TA-95M, fostering the development of more culturally aware music
foundation models.
CultureMERT-95M, a multi-culturally adapted foundation model, enhances cross-cultural music representation learning with a two-stage continual pre-training strategy, demonstrating superior performance in diverse non-Western music tasks.
Music foundation models audio representation learning CultureMERT-95M two-stage continual pre-training learning rate re-warming learning rate re-decaying multi-cultural data mix ROC-AUC AP non-Western music auto-tagging tasks task arithmetic single-culture adapted models cross-cultural evaluation CultureMERT-TA-95M