End-to-End Speech Recognition Contextualization with Large Language
  Models
In recent years, Large Language Models (LLMs) have garnered significant
attention from the research community due to their exceptional performance and
generalization capabilities. In this paper, we introduce a novel method for
contextualizing speech recognition models incorporating LLMs. Our approach
casts speech recognition as a mixed-modal language modeling task based on a
pretrained LLM. We provide audio features, along with optional text tokens for
context, to train the system to complete transcriptions in a decoder-only
fashion. As a result, the system is implicitly incentivized to learn how to
leverage unstructured contextual information during training. Our empirical
results demonstrate a significant improvement in performance, with a 6% WER
reduction when additional textual context is provided. Moreover, we find that
our method performs competitively and improve by 7.5% WER overall and 17% WER
on rare words against a baseline contextualized RNN-T system that has been
trained on more than twenty five times larger speech dataset. Overall, we
demonstrate that by only adding a handful number of trainable parameters via
adapters, we can unlock contextualized speech recognition capability for the
pretrained LLM while keeping the same text-only input functionality.
650bc56e2feef65026c8c52e
Egor Lakomkin
admin_assigned
2023-09-21T09:52:17.558Z
61e1bb4a210b4f412674d08d
/avatars/b0455785ca48ec93fc43a36dd4b5e417.svg
Egor Lakomkin
user
EgorL
650bc56e2feef65026c8c52f
Chunyang Wu
admin_assigned
2023-09-21T09:52:29.121Z
641d8fb516ef0e5a7274cf1a
/avatars/97a8565e7518f2a9b9e58dd4b98207d9.svg
wuchunyang
user
Picc
650bc56e2feef65026c8c530
Yassir Fathullah
admin_assigned
2023-09-21T09:52:36.351Z
64106057b27543634e3922bd
/avatars/c4eea7d419453802c67c4f7355315cac.svg
Yassir Fathullah
user
yassirf
650bc56e2feef65026c8c531
Ozlem Kalinli
650bc56e2feef65026c8c532
Michael L. Seltzer
650bc56e2feef65026c8c533
Christian Fuegen
End-to-End Speech Recognition Contextualization with Large Language
  Models
In recent years, Large Language Models (LLMs) have garnered significant
attention from the research community due to their exceptional performance and
generalization capabilities. In this paper, we introduce a novel method for
contextualizing speech recognition models incorporating LLMs. Our approach
casts speech recognition as a mixed-modal language modeling task based on a
pretrained LLM. We provide audio features, along with optional text tokens for
context, to train the system to complete transcriptions in a decoder-only
fashion. As a result, the system is implicitly incentivized to learn how to
leverage unstructured contextual information during training. Our empirical
results demonstrate a significant improvement in performance, with a 6% WER
reduction when additional textual context is provided. Moreover, we find that
our method performs competitively and improve by 7.5% WER overall and 17% WER
on rare words against a baseline contextualized RNN-T system that has been
trained on more than twenty five times larger speech dataset. Overall, we
demonstrate that by only adding a handful number of trainable parameters via
adapters, we can unlock contextualized speech recognition capability for the
pretrained LLM while keeping the same text-only input functionality.
A novel method using LLMs for contextualizing speech recognition models achieves significant performance improvements with minimal additional parameters.
Large Language Models (LLMs) mixed-modal language modeling pretrained LLMs decoder-only contextual information WER reduction contextualized RNN-T adapters contextualized speech recognition