title: CLaMP 3: Universal Music Information Retrieval Across Unaligned
    Modalities and Unseen Languages
summary: CLaMP 3 is a unified framework developed to address challenges of cross-modal
  and cross-lingual generalization in music information retrieval. Using
  contrastive learning, it aligns all major music modalities--including sheet
  music, performance signals, and audio recordings--with multilingual text in a
  shared representation space, enabling retrieval across unaligned modalities
  with text as a bridge. It features a multilingual text encoder adaptable to
  unseen languages, exhibiting strong cross-lingual generalization. Leveraging
  retrieval-augmented generation, we curated M4-RAG, a web-scale dataset
  consisting of 2.31 million music-text pairs. This dataset is enriched with
  detailed metadata that represents a wide array of global musical traditions. To
  advance future research, we release WikiMT-X, a benchmark comprising 1,000
  triplets of sheet music, audio, and richly varied text descriptions.
  Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple
  MIR tasks, significantly surpassing previous strong baselines and demonstrating
  excellent generalization in multimodal and multilingual music contexts.
paper_authors._id: 67b2e11dd2ee8e627dec1bc2
paper_authors.name: Shangda Wu
paper_authors._id: 67b2e11dd2ee8e627dec1bc3
paper_authors.name: Zhancheng Guo
paper_authors._id: 67b2e11dd2ee8e627dec1bc4
paper_authors.name: Ruibin Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:44:06.585Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 67b2e11dd2ee8e627dec1bc5
paper_authors.name: Junyan Jiang
paper_authors._id: 67b2e11dd2ee8e627dec1bc6
paper_authors.name: Seungheon Doh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:58:42.624Z
paper_authors._id: 637c3504c292c0fd3f37361f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637c3504c292c0fd3f37361f/wyTkbYKi8HufRT65LGN0P.jpeg
paper_authors.fullname: seungheon.doh
paper_authors.type: user
paper_authors.user: seungheondoh
paper_authors._id: 67b2e11dd2ee8e627dec1bc7
paper_authors.name: Gus Xia
paper_authors._id: 67b2e11dd2ee8e627dec1bc8
paper_authors.name: Juhan Nam
paper_authors._id: 67b2e11dd2ee8e627dec1bc9
paper_authors.name: Xiaobing Li
paper_authors._id: 67b2e11dd2ee8e627dec1bca
paper_authors.name: Feng Yu
paper_authors._id: 67b2e11dd2ee8e627dec1bcb
paper_authors.name: Maosong Sun
paper_title: CLaMP 3: Universal Music Information Retrieval Across Unaligned
    Modalities and Unseen Languages
paper_summary: CLaMP 3 is a unified framework developed to address challenges of cross-modal
  and cross-lingual generalization in music information retrieval. Using
  contrastive learning, it aligns all major music modalities--including sheet
  music, performance signals, and audio recordings--with multilingual text in a
  shared representation space, enabling retrieval across unaligned modalities
  with text as a bridge. It features a multilingual text encoder adaptable to
  unseen languages, exhibiting strong cross-lingual generalization. Leveraging
  retrieval-augmented generation, we curated M4-RAG, a web-scale dataset
  consisting of 2.31 million music-text pairs. This dataset is enriched with
  detailed metadata that represents a wide array of global musical traditions. To
  advance future research, we release WikiMT-X, a benchmark comprising 1,000
  triplets of sheet music, audio, and richly varied text descriptions.
  Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple
  MIR tasks, significantly surpassing previous strong baselines and demonstrating
  excellent generalization in multimodal and multilingual music contexts.
paper_ai_keywords: contrastive learning multilingual text encoder retrieval-augmented generation M4-RAG WikiMT-X sheet music audio recordings text descriptions MIR tasks