Comparing Discrete and Continuous Space LLMs for Speech Recognition
<|reference_start|>Comparing Discrete and Continuous Space LLMs for Speech Recognition: This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69\% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.<|reference_end|>
@article{xu2024comparing,
    title={Comparing Discrete and Continuous Space LLMs for Speech Recognition},
    author={Yaoxun Xu, Shi-Xiong Zhang, Jianwei Yu, Zhiyong Wu, Dong Yu},
    journal={arXiv preprint arXiv:2409.00800},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.00800},
    primaryClass={cs.CL}
}