A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR
<|reference_start|>A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR: Recent works have shown promising results in connecting speech encoders to large language models (LLMs) for speech recognition. However, several limitations persist, including limited fine-tuning options, a lack of mechanisms to enforce speech-text alignment, and high insertion errors especially in domain mismatch conditions. This paper presents a comprehensive solution to address these issues. We begin by investigating more thoughtful fine-tuning schemes. Next, we propose a matching loss to enhance alignment between modalities. Finally, we explore training and inference methods to mitigate high insertion errors. Experimental results on the Librispeech corpus demonstrate that partially fine-tuning the encoder and LLM using parameter-efficient methods, such as LoRA, is the most cost-effective approach. Additionally, the matching loss improves modality alignment, enhancing performance. The proposed training and inference methods significantly reduce insertion errors.<|reference_end|>
@article{pham2024a,
    title={A Comprehensive Solution to Connect Speech Encoder and Large Language
  Model for ASR},
    author={Van Tung Pham, Yist Lin, Tao Han, Wei Li, Jun Zhang, Lu Lu, Yuxuan
  Wang},
    journal={arXiv preprint arXiv:2406.17272},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17272},
    primaryClass={cs.LG}
}