CLaMP 3: Universal Music Information Retrieval Across Unaligned
  Modalities and Unseen Languages
CLaMP 3 is a unified framework developed to address challenges of cross-modal
and cross-lingual generalization in music information retrieval. Using
contrastive learning, it aligns all major music modalities--including sheet
music, performance signals, and audio recordings--with multilingual text in a
shared representation space, enabling retrieval across unaligned modalities
with text as a bridge. It features a multilingual text encoder adaptable to
unseen languages, exhibiting strong cross-lingual generalization. Leveraging
retrieval-augmented generation, we curated M4-RAG, a web-scale dataset
consisting of 2.31 million music-text pairs. This dataset is enriched with
detailed metadata that represents a wide array of global musical traditions. To
advance future research, we release WikiMT-X, a benchmark comprising 1,000
triplets of sheet music, audio, and richly varied text descriptions.
Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple
MIR tasks, significantly surpassing previous strong baselines and demonstrating
excellent generalization in multimodal and multilingual music contexts.
67b2e11dd2ee8e627dec1bc2
Shangda Wu
67b2e11dd2ee8e627dec1bc3
Zhancheng Guo
67b2e11dd2ee8e627dec1bc4
Ruibin Yuan
claimed_verified
2025-03-12T08:44:06.585Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
67b2e11dd2ee8e627dec1bc5
Junyan Jiang
67b2e11dd2ee8e627dec1bc6
Seungheon Doh
claimed_verified
2025-02-28T12:58:42.624Z
637c3504c292c0fd3f37361f
https://cdn-avatars.huggingface.co/v1/production/uploads/637c3504c292c0fd3f37361f/wyTkbYKi8HufRT65LGN0P.jpeg
seungheon.doh
user
seungheondoh
67b2e11dd2ee8e627dec1bc7
Gus Xia
67b2e11dd2ee8e627dec1bc8
Juhan Nam
67b2e11dd2ee8e627dec1bc9
Xiaobing Li
67b2e11dd2ee8e627dec1bca
Feng Yu
67b2e11dd2ee8e627dec1bcb
Maosong Sun
CLaMP 3: Universal Music Information Retrieval Across Unaligned
  Modalities and Unseen Languages
CLaMP 3 is a unified framework developed to address challenges of cross-modal
and cross-lingual generalization in music information retrieval. Using
contrastive learning, it aligns all major music modalities--including sheet
music, performance signals, and audio recordings--with multilingual text in a
shared representation space, enabling retrieval across unaligned modalities
with text as a bridge. It features a multilingual text encoder adaptable to
unseen languages, exhibiting strong cross-lingual generalization. Leveraging
retrieval-augmented generation, we curated M4-RAG, a web-scale dataset
consisting of 2.31 million music-text pairs. This dataset is enriched with
detailed metadata that represents a wide array of global musical traditions. To
advance future research, we release WikiMT-X, a benchmark comprising 1,000
triplets of sheet music, audio, and richly varied text descriptions.
Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple
MIR tasks, significantly surpassing previous strong baselines and demonstrating
excellent generalization in multimodal and multilingual music contexts.
CLaMP 3 uses contrastive learning to align music modalities with multilingual text, enabling effective cross-modal and cross-lingual retrieval and generation in music information retrieval.
contrastive learning sheet music performance signals audio recordings multilingual text encoder retrieval-augmented generation M4-RAG WikiMT-X multimodal multilingual music