title: Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for
    Speech Understanding
summary: Large Language Models (LLMs) have been applied in the speech domain, often
  incurring a performance drop due to misaligned between speech and language
  representations. To bridge this gap, we propose a joint speech and language
  model (SLM) using a Speech2Text adapter, which maps speech into text token
  embedding space without speech information loss. Additionally, using a
  CTC-based blank-filtering, we can reduce the speech sequence length to that of
  text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the
  dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to
  address errors on rare entities, we augment SLM with a Speech2Entity retriever,
  which uses speech to retrieve relevant entities, and then adds them to the
  original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the
  DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with
  the dialog understanding task improves the ASR performance from 9.4% to 8.5%
  WER.
paper_authors._id: 648921ae47b1bd57a66bca1f
paper_authors.name: Mingqiu Wang
paper_authors._id: 648921ae47b1bd57a66bca20
paper_authors.name: Izhak Shafran
paper_authors._id: 648921ae47b1bd57a66bca21
paper_authors.name: Hagen Soltau
paper_authors._id: 648921ae47b1bd57a66bca22
paper_authors.name: Wei Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T20:37:16.101Z
paper_authors._id: 6489e3992f9e11a0c26973b6
paper_authors.avatarUrl: /avatars/e637e6caab6df56da6005f73c613a3d0.svg
paper_authors.fullname: Wei Han
paper_authors.type: user
paper_authors.user: weihan3
paper_authors._id: 648921ae47b1bd57a66bca23
paper_authors.name: Yuan Cao
paper_authors._id: 648921ae47b1bd57a66bca24
paper_authors.name: Dian Yu
paper_authors._id: 648921ae47b1bd57a66bca25
paper_authors.name: Laurent El Shafey
paper_title: Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for
    Speech Understanding
paper_summary: Large Language Models (LLMs) have been applied in the speech domain, often
  incurring a performance drop due to misaligned between speech and language
  representations. To bridge this gap, we propose a joint speech and language
  model (SLM) using a Speech2Text adapter, which maps speech into text token
  embedding space without speech information loss. Additionally, using a
  CTC-based blank-filtering, we can reduce the speech sequence length to that of
  text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the
  dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to
  address errors on rare entities, we augment SLM with a Speech2Entity retriever,
  which uses speech to retrieve relevant entities, and then adds them to the
  original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the
  DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with
  the dialog understanding task improves the ASR performance from 9.4% to 8.5%
  WER.
paper_ai_keywords: large language models (LLMs) speech domain speech and language model (SLM) Speech2Text adapter text token embedding space CTC-based blank-filtering speech sequence length text sequence length speech MultiWoz dataset (DSTC11) dialog state tracking (DST) accuracy retrieval-augmented SLM (ReSLM) rare entities Speech2Entity retriever WER ASR task dialog understanding task