Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection
<|reference_start|>Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection: As a robust and large-scale multilingual speech recognition model, Whisper has demonstrated impressive results in many low-resource and out-of-distribution scenarios. However, its encoder-decoder structure hinders its application to streaming speech recognition. In this paper, we introduce Simul-Whisper, which uses the time alignment embedded in Whisper's cross-attention to guide auto-regressive decoding and achieve chunk-based streaming ASR without any fine-tuning of the pre-trained model. Furthermore, we observe the negative effect of the truncated words at the chunk boundaries on the decoding results and propose an integrate-and-fire-based truncation detection model to address this issue. Experiments on multiple languages and Whisper architectures show that Simul-Whisper achieves an average absolute word error rate degradation of only 1.46% at a chunk size of 1 second, which significantly outperforms the current state-of-the-art baseline.<|reference_end|>
@article{wang2024simul-whisper:,
    title={Simul-Whisper: Attention-Guided Streaming Whisper with Truncation
  Detection},
    author={Haoyu Wang, Guoqiang Hu, Guodong Lin, Wei-Qiang Zhang, Jian Li},
    journal={arXiv preprint arXiv:2406.10052},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10052},
    primaryClass={cs.SD cs.CL eess.AS}
}