ML-SUPERB 20: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets
<|reference_start|>ML-SUPERB 20: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets: ML-SUPERB evaluates self-supervised learning (SSL) models on the tasks of language identification and automatic speech recognition (ASR). This benchmark treats the models as feature extractors and uses a single shallow downstream model, which can be fine-tuned for a downstream task. However, real-world use cases may require different configurations. This paper presents ML-SUPERB~2.0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models across downstream models, fine-tuning setups, and efficient model adaptation approaches. We find performance improvements over the setup of ML-SUPERB. However, performance depends on the downstream model design. Also, we find large performance differences between languages and datasets, suggesting the need for more targeted approaches to improve multilingual ASR performance.<|reference_end|>
@article{shi2024ml-superb,
    title={ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling
  Constraints, Languages, and Datasets},
    author={Jiatong Shi, Shih-Heng Wang, William Chen, Martijn Bartelds, Vanya
  Bannihatti Kumar, Jinchuan Tian, Xuankai Chang, Dan Jurafsky, Karen Livescu,
  Hung-yi Lee, and Shinji Watanabe},
    journal={arXiv preprint arXiv:2406.08641},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.08641},
    primaryClass={cs.SD cs.CL eess.AS}
}