ASR Benchmarking: Need for a More Representative Conversational Dataset
<|reference_start|>ASR Benchmarking: Need for a More Representative Conversational Dataset: Automatic Speech Recognition (ASR) systems have achieved remarkable performance on widely used benchmarks such as LibriSpeech and Fleurs. However, these benchmarks do not adequately reflect the complexities of real-world conversational environments, where speech is often unstructured and contains disfluencies such as pauses, interruptions, and diverse accents. In this study, we introduce a multilingual conversational dataset, derived from TalkBank, consisting of unstructured phone conversation between adults. Our results show a significant performance drop across various state-of-the-art ASR models when tested in conversational settings. Furthermore, we observe a correlation between Word Error Rate and the presence of speech disfluencies, highlighting the critical need for more realistic, conversational ASR benchmarks.<|reference_end|>
@article{maheshwari2024asr,
    title={ASR Benchmarking: Need for a More Representative Conversational Dataset},
    author={Gaurav Maheshwari, Dmitry Ivanov, Th'eo Johannet, Kevin El Haddad},
    journal={arXiv preprint arXiv:2409.12042},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.12042},
    primaryClass={cs.CL cs.SD eess.AS}
}