Contextualization of ASR with LLM using phonetic retrieval-based augmentation
<|reference_start|>Contextualization of ASR with LLM using phonetic retrieval-based augmentation: Large language models (LLMs) have shown superb capability of modeling multimodal signals including audio and text, allowing the model to generate spoken or textual response given a speech input. However, it remains a challenge for the model to recognize personal named entities, such as contacts in a phone book, when the input modality is speech. In this work, we start with a speech recognition task and propose a retrieval-based solution to contextualize the LLM: we first let the LLM detect named entities in speech without any context, then use this named entity as a query to retrieve phonetically similar named entities from a personal database and feed them to the LLM, and finally run context-aware LLM decoding. In a voice assistant task, our solution achieved up to 30.2% relative word error rate reduction and 73.6% relative named entity error rate reduction compared to a baseline system without contextualization. Notably, our solution by design avoids prompting the LLM with the full named entity database, making it highly efficient and applicable to large named entity databases.<|reference_end|>
@article{lei2024contextualization,
    title={Contextualization of ASR with LLM using phonetic retrieval-based
  augmentation},
    author={Zhihong Lei, Xingyu Na, Mingbin Xu, Ernest Pusateri, Christophe Van
  Gysel, Yuanyuan Zhang, Shiyi Han and Zhen Huang},
    journal={arXiv preprint arXiv:2409.15353},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.15353},
    primaryClass={eess.AS cs.CL cs.LG cs.SD}
}