title: Towards Robust Speech Representation Learning for Thousands of Languages
summary: Self-supervised learning (SSL) has helped extend speech technologies to more
  languages by reducing the need for labeled data. However, models are still far
  from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual
  Encoder for Universal Speech, trained on over 1 million hours of data across
  4057 languages, extending the language coverage of SSL models 4-fold. We
  combine 1 million hours of speech from existing publicly accessible corpora
  with a newly created corpus of 7400+ hours from 4057 languages, which will be
  publicly released. To handle the diverse conditions of multilingual speech
  data, we augment the typical SSL masked prediction approach with a novel
  dereverberation objective, increasing robustness. We evaluate XEUS on several
  benchmarks, and show that it consistently outperforms or achieves comparable
  results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS
  sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT
  2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or
  pre-training data. Checkpoints, code, and data are found in
  https://www.wavlab.org/activities/2024/xeus/.
paper_authors._id: 668390c4c7d9ff0922172a12
paper_authors.name: William Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-02T05:31:49.168Z
paper_authors._id: 630438615c70c21d0eae6613
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630438615c70c21d0eae6613/KEHsB-znDIKHt7M0eyP8j.png
paper_authors.fullname: William Chen
paper_authors.type: user
paper_authors.user: wanchichen
paper_authors._id: 668390c4c7d9ff0922172a13
paper_authors.name: Wangyou Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:20:38.342Z
paper_authors._id: 6205e8ab6c4c35a0a2415c81
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644558391919-6205e8ab6c4c35a0a2415c81.png
paper_authors.fullname: Wangyou Zhang
paper_authors.type: user
paper_authors.user: wyz
paper_authors._id: 668390c4c7d9ff0922172a14
paper_authors.name: Yifan Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:20:44.860Z
paper_authors._id: 61809f31a367a8f5351ef353
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61809f31a367a8f5351ef353/s5eQ00YeoirakzE_rJ0cy.jpeg
paper_authors.fullname: Yifan Peng
paper_authors.type: user
paper_authors.user: pyf98
paper_authors._id: 668390c4c7d9ff0922172a15
paper_authors.name: Xinjian Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:20:57.432Z
paper_authors._id: 648cd0277f7821f063c16e9a
paper_authors.avatarUrl: /avatars/cb3d7745f22010d8a3f1b6a28975a561.svg
paper_authors.fullname: Xinjian Li
paper_authors.type: user
paper_authors.user: xinjianl
paper_authors._id: 668390c4c7d9ff0922172a16
paper_authors.name: Jinchuan Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:21:03.671Z
paper_authors._id: 64770ff6d7cf39f2e937b882
paper_authors.avatarUrl: /avatars/7b729501856f5073cf9e484811012f2a.svg
paper_authors.fullname: Jinchuan Tian
paper_authors.type: user
paper_authors.user: JinchuanTian
paper_authors._id: 668390c4c7d9ff0922172a17
paper_authors.name: Jiatong Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:21:09.544Z
paper_authors._id: 6136ec7a40e43b8f748a0832
paper_authors.avatarUrl: /avatars/089afa82ecfe6432d2dbfe81048c6ffa.svg
paper_authors.fullname: Jiatong Shi
paper_authors.type: user
paper_authors.user: ftshijt
paper_authors._id: 668390c4c7d9ff0922172a18
paper_authors.name: Xuankai Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:21:15.845Z
paper_authors._id: 6136b5707c6d9b8e41469bec
paper_authors.avatarUrl: /avatars/5b7f4ee0ba36bb6b87accda0ee27bb17.svg
paper_authors.fullname: Xuankai Chang
paper_authors.type: user
paper_authors.user: simpleoier
paper_authors._id: 668390c4c7d9ff0922172a19
paper_authors.name: Soumi Maiti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:21:21.415Z
paper_authors._id: 6331d6465102481d8dd2b55e
paper_authors.avatarUrl: /avatars/6dc792023b896e526d636c9c5d6e6547.svg
paper_authors.fullname: Soumi Maiti
paper_authors.type: user
paper_authors.user: soumi-maiti
paper_authors._id: 668390c4c7d9ff0922172a1a
paper_authors.name: Karen Livescu
paper_authors._id: 668390c4c7d9ff0922172a1b
paper_authors.name: Shinji Watanabe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:21:47.692Z
paper_authors._id: 5feb9dbc29bcc7d2abb1cb76
paper_authors.avatarUrl: /avatars/908d392634377e674ba9e5a4318b101e.svg
paper_authors.fullname: Shinji Watanabe
paper_authors.type: user
paper_authors.user: sw005320
paper_title: Towards Robust Speech Representation Learning for Thousands of Languages
paper_summary: Self-supervised learning (SSL) has helped extend speech technologies to more
  languages by reducing the need for labeled data. However, models are still far
  from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual
  Encoder for Universal Speech, trained on over 1 million hours of data across
  4057 languages, extending the language coverage of SSL models 4-fold. We
  combine 1 million hours of speech from existing publicly accessible corpora
  with a newly created corpus of 7400+ hours from 4057 languages, which will be
  publicly released. To handle the diverse conditions of multilingual speech
  data, we augment the typical SSL masked prediction approach with a novel
  dereverberation objective, increasing robustness. We evaluate XEUS on several
  benchmarks, and show that it consistently outperforms or achieves comparable
  results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS
  sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT
  2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or
  pre-training data. Checkpoints, code, and data are found in
  https://www.wavlab.org/activities/2024/xeus/.
paper_ai_summary: XEUS, a cross-lingual encoder for universal speech, enhances language coverage and robustness through novel dereverberation, outperforming existing models on various benchmarks with less data and parameters.
paper_ai_keywords: self-supervised learning cross-lingual encoder SSL masked prediction dereverberation language coverage multilingual speech data ML-SUPERB benchmark MMS 1B w2v-BERT 2.0 v2