Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study
<|reference_start|>Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study: Unified speech-text models like SpeechGPT, VioLA, and AudioPaLM have shown impressive performance across various speech-related tasks, especially in Automatic Speech Recognition (ASR). These models typically adopt a unified method to model discrete speech and text tokens, followed by training a decoder-only transformer. However, they are all designed for non-streaming ASR tasks, where the entire speech utterance is needed during decoding. Hence, we introduce a decoder-only model exclusively designed for streaming recognition, incorporating a dedicated boundary token to facilitate streaming recognition and employing causal attention masking during the training phase. Furthermore, we introduce right-chunk attention and various data augmentation techniques to improve the model's contextual modeling abilities. While achieving streaming speech recognition, experiments on the AISHELL-1 and -2 datasets demonstrate the competitive performance of our streaming approach with non-streaming decoder-only counterparts.<|reference_end|>
@article{chen2024streaming,
    title={Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech
  Units: A Pilot Study},
    author={Peikun Chen, Sining Sun, Changhao Shan, Qing Yang, Lei Xie},
    journal={arXiv preprint arXiv:2406.18862},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.18862},
    primaryClass={cs.SD eess.AS}
}