Low-rank Adaptation of Large Language Model Rescoring for
  Parameter-Efficient Speech Recognition
We propose a neural language modeling system based on low-rank adaptation
(LoRA) for speech recognition output rescoring. Although pretrained language
models (LMs) like BERT have shown superior performance in second-pass
rescoring, the high computational cost of scaling up the pretraining stage and
adapting the pretrained models to specific domains limit their practical use in
rescoring. Here we present a method based on low-rank decomposition to train a
rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
of the pretrained parameters. These inserted matrices are optimized through a
discriminative training objective along with a correlation-based regularization
loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
evaluated on LibriSpeech and internal datasets with decreased training times by
factors between 5.4 and 3.6.
6514ef2fd1005268b0c639e2
Yu Yu
6514ef2fd1005268b0c639e3
Chao-Han Huck Yang
claimed_verified
2023-10-02T07:00:20.617Z
629e1b71bb6419817ed7566c
https://cdn-avatars.huggingface.co/v1/production/uploads/629e1b71bb6419817ed7566c/0ZCt-11eQtRDCOk9AozOp.jpeg
Huck Yang
user
huckiyang
6514ef2fd1005268b0c639e4
Jari Kolehmainen
6514ef2fd1005268b0c639e5
Prashanth G. Shivakumar
6514ef2fd1005268b0c639e6
Yile Gu
6514ef2fd1005268b0c639e7
Sungho Ryu
6514ef2fd1005268b0c639e8
Roger Ren
6514ef2fd1005268b0c639e9
Qi Luo
6514ef2fd1005268b0c639ea
Aditya Gourav
6514ef2fd1005268b0c639eb
I-Fan Chen
admin_assigned
2023-09-28T09:29:53.893Z
6352653c507b679c3c5c4c2b
/avatars/9300329ac2861571b04bbbf3ce2ca195.svg
I FAN CHEN
user
ddven
6514ef2fd1005268b0c639ec
Yi-Chieh Liu
6514ef2fd1005268b0c639ed
Tuan Dinh
6514ef2fd1005268b0c639ee
Ankur Gandhe
6514ef2fd1005268b0c639ef
Denis Filimonov
6514ef2fd1005268b0c639f0
Shalini Ghosh
6514ef2fd1005268b0c639f1
Andreas Stolcke
6514ef2fd1005268b0c639f2
Ariya Rastow
6514ef2fd1005268b0c639f3
Ivan Bulyko
Low-rank Adaptation of Large Language Model Rescoring for
  Parameter-Efficient Speech Recognition
We propose a neural language modeling system based on low-rank adaptation
(LoRA) for speech recognition output rescoring. Although pretrained language
models (LMs) like BERT have shown superior performance in second-pass
rescoring, the high computational cost of scaling up the pretraining stage and
adapting the pretrained models to specific domains limit their practical use in
rescoring. Here we present a method based on low-rank decomposition to train a
rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
of the pretrained parameters. These inserted matrices are optimized through a
discriminative training objective along with a correlation-based regularization
loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
evaluated on LibriSpeech and internal datasets with decreased training times by
factors between 5.4 and 3.6.
Rescore-BERT, using low-rank adaptation, achieves efficient domain-specific speech recognition output rescoring with reduced training parameters and time.
low-rank adaptation LoRA pretrained language models BERT LibriSpeech discriminative training objective correlation-based regularization loss