Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM
<|reference_start|>Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM: Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research.<|reference_end|>
@article{yuen2024internalizing,
    title={Internalizing ASR with Implicit Chain of Thought for Efficient
  Speech-to-Speech Conversational LLM},
    author={Robin Shing-Hei Yuen, Timothy Tin-Long Tse, Jian Zhu},
    journal={arXiv preprint arXiv:2409.17353},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17353},
    primaryClass={cs.CL}
}