Faster Speech-LLaMA Inference with Multi-token Prediction
<|reference_start|>Faster Speech-LLaMA Inference with Multi-token Prediction: Large language models (LLMs) have become proficient at solving a wide variety of tasks, including those involving multi-modal inputs. In particular, instantiating an LLM (such as LLaMA) with a speech encoder and training it on paired data imparts speech recognition (ASR) abilities to the decoder-only model, hence called Speech-LLaMA. Nevertheless, due to the sequential nature of auto-regressive inference and the relatively large decoder, Speech-LLaMA models require relatively high inference time. In this work, we propose to speed up Speech-LLaMA inference by predicting multiple tokens in the same decoding step. We explore several model architectures that enable this, and investigate their performance using threshold-based and verification-based inference strategies. We also propose a prefix-based beam search decoding method that allows efficient minimum word error rate (MWER) training for such models. We evaluate our models on a variety of public benchmarks, where they reduce the number of decoder calls by ~3.2x while maintaining or improving WER performance.<|reference_end|>
@article{raj2024faster,
    title={Faster Speech-LLaMA Inference with Multi-token Prediction},
    author={Desh Raj, Gil Keren, Junteng Jia, Jay Mahadeokar, Ozlem Kalinli},
    journal={arXiv preprint arXiv:2409.08148},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.08148},
    primaryClass={eess.AS cs.SD}
}