Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for
  Speech Understanding
Large Language Models (LLMs) have been applied in the speech domain, often
incurring a performance drop due to misaligned between speech and language
representations. To bridge this gap, we propose a joint speech and language
model (SLM) using a Speech2Text adapter, which maps speech into text token
embedding space without speech information loss. Additionally, using a
CTC-based blank-filtering, we can reduce the speech sequence length to that of
text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the
dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to
address errors on rare entities, we augment SLM with a Speech2Entity retriever,
which uses speech to retrieve relevant entities, and then adds them to the
original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the
DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with
the dialog understanding task improves the ASR performance from 9.4% to 8.5%
WER.
648921ae47b1bd57a66bca1f
Mingqiu Wang
648921ae47b1bd57a66bca20
Izhak Shafran
648921ae47b1bd57a66bca21
Hagen Soltau
648921ae47b1bd57a66bca22
Wei Han
claimed_verified
2023-06-14T20:37:16.101Z
6489e3992f9e11a0c26973b6
/avatars/e637e6caab6df56da6005f73c613a3d0.svg
Wei Han
user
weihan3
648921ae47b1bd57a66bca23
Yuan Cao
648921ae47b1bd57a66bca24
Dian Yu
648921ae47b1bd57a66bca25
Laurent El Shafey
Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for
  Speech Understanding
Large Language Models (LLMs) have been applied in the speech domain, often
incurring a performance drop due to misaligned between speech and language
representations. To bridge this gap, we propose a joint speech and language
model (SLM) using a Speech2Text adapter, which maps speech into text token
embedding space without speech information loss. Additionally, using a
CTC-based blank-filtering, we can reduce the speech sequence length to that of
text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the
dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to
address errors on rare entities, we augment SLM with a Speech2Entity retriever,
which uses speech to retrieve relevant entities, and then adds them to the
original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the
DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with
the dialog understanding task improves the ASR performance from 9.4% to 8.5%
WER.
A joint speech and language model using a Speech2Text adapter and CTC-based blank-filtering improves dialog state tracking and ASR performance in the speech domain.
Speech2Text adapter CTC-based blank-filtering dialog state tracking (DST) Speech2Entity retriever retrieval-augmented SLM (ReSLM) Automatic Speech Recognition (ASR) Word Error Rate (WER)