title: Low-rank Adaptation of Large Language Model Rescoring for
    Parameter-Efficient Speech Recognition
summary: We propose a neural language modeling system based on low-rank adaptation
  (LoRA) for speech recognition output rescoring. Although pretrained language
  models (LMs) like BERT have shown superior performance in second-pass
  rescoring, the high computational cost of scaling up the pretraining stage and
  adapting the pretrained models to specific domains limit their practical use in
  rescoring. Here we present a method based on low-rank decomposition to train a
  rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
  of the pretrained parameters. These inserted matrices are optimized through a
  discriminative training objective along with a correlation-based regularization
  loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
  evaluated on LibriSpeech and internal datasets with decreased training times by
  factors between 5.4 and 3.6.
paper_authors._id: 6514ef2fd1005268b0c639e2
paper_authors.name: Yu Yu
paper_authors._id: 6514ef2fd1005268b0c639e3
paper_authors.name: Chao-Han Huck Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-02T07:00:20.617Z
paper_authors._id: 629e1b71bb6419817ed7566c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629e1b71bb6419817ed7566c/0ZCt-11eQtRDCOk9AozOp.jpeg
paper_authors.fullname: Huck Yang
paper_authors.type: user
paper_authors.user: huckiyang
paper_authors._id: 6514ef2fd1005268b0c639e4
paper_authors.name: Jari Kolehmainen
paper_authors._id: 6514ef2fd1005268b0c639e5
paper_authors.name: Prashanth G. Shivakumar
paper_authors._id: 6514ef2fd1005268b0c639e6
paper_authors.name: Yile Gu
paper_authors._id: 6514ef2fd1005268b0c639e7
paper_authors.name: Sungho Ryu
paper_authors._id: 6514ef2fd1005268b0c639e8
paper_authors.name: Roger Ren
paper_authors._id: 6514ef2fd1005268b0c639e9
paper_authors.name: Qi Luo
paper_authors._id: 6514ef2fd1005268b0c639ea
paper_authors.name: Aditya Gourav
paper_authors._id: 6514ef2fd1005268b0c639eb
paper_authors.name: I-Fan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T09:29:53.893Z
paper_authors._id: 6352653c507b679c3c5c4c2b
paper_authors.avatarUrl: /avatars/9300329ac2861571b04bbbf3ce2ca195.svg
paper_authors.fullname: I FAN CHEN
paper_authors.type: user
paper_authors.user: ddven
paper_authors._id: 6514ef2fd1005268b0c639ec
paper_authors.name: Yi-Chieh Liu
paper_authors._id: 6514ef2fd1005268b0c639ed
paper_authors.name: Tuan Dinh
paper_authors._id: 6514ef2fd1005268b0c639ee
paper_authors.name: Ankur Gandhe
paper_authors._id: 6514ef2fd1005268b0c639ef
paper_authors.name: Denis Filimonov
paper_authors._id: 6514ef2fd1005268b0c639f0
paper_authors.name: Shalini Ghosh
paper_authors._id: 6514ef2fd1005268b0c639f1
paper_authors.name: Andreas Stolcke
paper_authors._id: 6514ef2fd1005268b0c639f2
paper_authors.name: Ariya Rastow
paper_authors._id: 6514ef2fd1005268b0c639f3
paper_authors.name: Ivan Bulyko
paper_title: Low-rank Adaptation of Large Language Model Rescoring for
    Parameter-Efficient Speech Recognition
paper_summary: We propose a neural language modeling system based on low-rank adaptation
  (LoRA) for speech recognition output rescoring. Although pretrained language
  models (LMs) like BERT have shown superior performance in second-pass
  rescoring, the high computational cost of scaling up the pretraining stage and
  adapting the pretrained models to specific domains limit their practical use in
  rescoring. Here we present a method based on low-rank decomposition to train a
  rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
  of the pretrained parameters. These inserted matrices are optimized through a
  discriminative training objective along with a correlation-based regularization
  loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
  evaluated on LibriSpeech and internal datasets with decreased training times by
  factors between 5.4 and 3.6.
paper_ai_keywords: low-rank adaptation (LoRA) neural language modeling pretrained language models (LMs) BERT low-rank decomposition rescoring BERT model discriminative training objective correlation-based regularization loss Rescore-BERT (LoRB) LibriSpeech internal datasets