title: End-to-End Speech Recognition Contextualization with Large Language
    Models
summary: In recent years, Large Language Models (LLMs) have garnered significant
  attention from the research community due to their exceptional performance and
  generalization capabilities. In this paper, we introduce a novel method for
  contextualizing speech recognition models incorporating LLMs. Our approach
  casts speech recognition as a mixed-modal language modeling task based on a
  pretrained LLM. We provide audio features, along with optional text tokens for
  context, to train the system to complete transcriptions in a decoder-only
  fashion. As a result, the system is implicitly incentivized to learn how to
  leverage unstructured contextual information during training. Our empirical
  results demonstrate a significant improvement in performance, with a 6% WER
  reduction when additional textual context is provided. Moreover, we find that
  our method performs competitively and improve by 7.5% WER overall and 17% WER
  on rare words against a baseline contextualized RNN-T system that has been
  trained on more than twenty five times larger speech dataset. Overall, we
  demonstrate that by only adding a handful number of trainable parameters via
  adapters, we can unlock contextualized speech recognition capability for the
  pretrained LLM while keeping the same text-only input functionality.
paper_authors._id: 650bc56e2feef65026c8c52e
paper_authors.name: Egor Lakomkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:52:17.558Z
paper_authors._id: 61e1bb4a210b4f412674d08d
paper_authors.avatarUrl: /avatars/b0455785ca48ec93fc43a36dd4b5e417.svg
paper_authors.fullname: Egor Lakomkin
paper_authors.type: user
paper_authors.user: EgorL
paper_authors._id: 650bc56e2feef65026c8c52f
paper_authors.name: Chunyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:52:29.121Z
paper_authors._id: 641d8fb516ef0e5a7274cf1a
paper_authors.avatarUrl: /avatars/97a8565e7518f2a9b9e58dd4b98207d9.svg
paper_authors.fullname: wuchunyang
paper_authors.type: user
paper_authors.user: Picc
paper_authors._id: 650bc56e2feef65026c8c530
paper_authors.name: Yassir Fathullah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:52:36.351Z
paper_authors._id: 64106057b27543634e3922bd
paper_authors.avatarUrl: /avatars/c4eea7d419453802c67c4f7355315cac.svg
paper_authors.fullname: Yassir Fathullah
paper_authors.type: user
paper_authors.user: yassirf
paper_authors._id: 650bc56e2feef65026c8c531
paper_authors.name: Ozlem Kalinli
paper_authors._id: 650bc56e2feef65026c8c532
paper_authors.name: Michael L. Seltzer
paper_authors._id: 650bc56e2feef65026c8c533
paper_authors.name: Christian Fuegen
paper_title: End-to-End Speech Recognition Contextualization with Large Language
    Models
paper_summary: In recent years, Large Language Models (LLMs) have garnered significant
  attention from the research community due to their exceptional performance and
  generalization capabilities. In this paper, we introduce a novel method for
  contextualizing speech recognition models incorporating LLMs. Our approach
  casts speech recognition as a mixed-modal language modeling task based on a
  pretrained LLM. We provide audio features, along with optional text tokens for
  context, to train the system to complete transcriptions in a decoder-only
  fashion. As a result, the system is implicitly incentivized to learn how to
  leverage unstructured contextual information during training. Our empirical
  results demonstrate a significant improvement in performance, with a 6% WER
  reduction when additional textual context is provided. Moreover, we find that
  our method performs competitively and improve by 7.5% WER overall and 17% WER
  on rare words against a baseline contextualized RNN-T system that has been
  trained on more than twenty five times larger speech dataset. Overall, we
  demonstrate that by only adding a handful number of trainable parameters via
  adapters, we can unlock contextualized speech recognition capability for the
  pretrained LLM while keeping the same text-only input functionality.
paper_ai_summary: A novel method using LLMs for contextualizing speech recognition models achieves significant performance improvements with minimal additional parameters.
paper_ai_keywords: Large Language Models (LLMs) mixed-modal language modeling pretrained LLMs decoder-only contextual information WER reduction contextualized RNN-T adapters contextualized speech recognition