Advancing Multi-talker ASR Performance with Large Language Models
<|reference_start|>Advancing Multi-talker ASR Performance with Large Language Models: Recognizing overlapping speech from multiple speakers in conversational scenarios is one of the most challenging problem for automatic speech recognition (ASR). Serialized output training (SOT) is a classic method to address multi-talker ASR, with the idea of concatenating transcriptions from multiple speakers according to the emission times of their speech for training. However, SOT-style transcriptions, derived from concatenating multiple related utterances in a conversation, depend significantly on modeling long contexts. Therefore, compared to traditional methods that primarily emphasize encoder performance in attention-based encoder-decoder (AED) architectures, a novel approach utilizing large language models (LLMs) that leverages the capabilities of pre-trained decoders may be better suited for such complex and challenging scenarios. In this paper, we propose an LLM-based SOT approach for multi-talker ASR, leveraging pre-trained speech encoder and LLM, fine-tuning them on multi-talker dataset using appropriate strategies. Experimental results demonstrate that our approach surpasses traditional AED-based methods on the simulated dataset LibriMix and achieves state-of-the-art performance on the evaluation set of the real-world dataset AMI, outperforming the AED model trained with 1000 times more supervised data in previous works.<|reference_end|>
@article{shi2024advancing,
    title={Advancing Multi-talker ASR Performance with Large Language Models},
    author={Mohan Shi, Zengrui Jin, Yaoxun Xu, Yong Xu, Shi-Xiong Zhang, Kun Wei,
  Yiwen Shao, Chunlei Zhang, Dong Yu},
    journal={arXiv preprint arXiv:2408.17431},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.17431},
    primaryClass={eess.AS cs.AI}
}