title: Denoising LM: Pushing the Limits of Error Correction Models for Speech
    Recognition
summary: Language models (LMs) have long been used to improve results of automatic
  speech recognition (ASR) systems, but they are unaware of the errors that ASR
  systems make. Error correction models are designed to fix ASR errors, however,
  they showed little improvement over traditional LMs mainly due to the lack of
  supervised training data. In this paper, we present Denoising LM (DLM), which
  is a scaled error correction model trained with vast amounts of
  synthetic data, significantly exceeding prior attempts meanwhile achieving new
  state-of-the-art ASR performance. We use text-to-speech (TTS) systems to
  synthesize audio, which is fed into an ASR system to produce noisy hypotheses,
  which are then paired with the original texts to train the DLM. DLM has several
  key ingredients: (i) up-scaled model and data; (ii) usage of
  multi-speaker TTS systems; (iii) combination of multiple noise augmentation
  strategies; and (iv) new decoding techniques. With a Transformer-CTC ASR, DLM
  achieves 1.5% word error rate (WER) on test-clean and 3.3% WER on
  test-other on Librispeech, which to our knowledge are the best
  reported numbers in the setting where no external audio data are used and even
  match self-supervised methods which use external audio data. Furthermore, a
  single DLM is applicable to different ASRs, and greatly surpassing the
  performance of conventional LM based beam-search rescoring. These results
  indicate that properly investigated error correction models have the potential
  to replace conventional LMs, holding the key to a new level of accuracy in ASR
  systems.
paper_authors._id: 6653eee1edd332b9443d2ada
paper_authors.name: Zijin Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:50:52.499Z
paper_authors._id: 6622d179ef00d8ae7a991bc1
paper_authors.avatarUrl: /avatars/8dfcdc14605cc47664c804b0ba5fc444.svg
paper_authors.fullname: Zijin Gu
paper_authors.type: user
paper_authors.user: lelegu
paper_authors._id: 6653eee1edd332b9443d2adb
paper_authors.name: Tatiana Likhomanenko
paper_authors._id: 6653eee1edd332b9443d2adc
paper_authors.name: He Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:19:14.964Z
paper_authors._id: 5f024bd8ff69163f6f59e775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652111948107-5f024bd8ff69163f6f59e775.png
paper_authors.fullname: Richard He Bai
paper_authors.type: user
paper_authors.user: richardbaihe
paper_authors._id: 6653eee1edd332b9443d2add
paper_authors.name: Erik McDermott
paper_authors._id: 6653eee1edd332b9443d2ade
paper_authors.name: Ronan Collobert
paper_authors._id: 6653eee1edd332b9443d2adf
paper_authors.name: Navdeep Jaitly
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:51:31.399Z
paper_authors._id: 631119df88942700629f4aab
paper_authors.avatarUrl: /avatars/251c20a8fd48831a8c0cc0d198a42455.svg
paper_authors.fullname: Navdeep Jaitly
paper_authors.type: user
paper_authors.user: ndjaitly
paper_title: Denoising LM: Pushing the Limits of Error Correction Models for Speech
    Recognition
paper_summary: Language models (LMs) have long been used to improve results of automatic
  speech recognition (ASR) systems, but they are unaware of the errors that ASR
  systems make. Error correction models are designed to fix ASR errors, however,
  they showed little improvement over traditional LMs mainly due to the lack of
  supervised training data. In this paper, we present Denoising LM (DLM), which
  is a scaled error correction model trained with vast amounts of
  synthetic data, significantly exceeding prior attempts meanwhile achieving new
  state-of-the-art ASR performance. We use text-to-speech (TTS) systems to
  synthesize audio, which is fed into an ASR system to produce noisy hypotheses,
  which are then paired with the original texts to train the DLM. DLM has several
  key ingredients: (i) up-scaled model and data; (ii) usage of
  multi-speaker TTS systems; (iii) combination of multiple noise augmentation
  strategies; and (iv) new decoding techniques. With a Transformer-CTC ASR, DLM
  achieves 1.5% word error rate (WER) on test-clean and 3.3% WER on
  test-other on Librispeech, which to our knowledge are the best
  reported numbers in the setting where no external audio data are used and even
  match self-supervised methods which use external audio data. Furthermore, a
  single DLM is applicable to different ASRs, and greatly surpassing the
  performance of conventional LM based beam-search rescoring. These results
  indicate that properly investigated error correction models have the potential
  to replace conventional LMs, holding the key to a new level of accuracy in ASR
  systems.
paper_ai_summary: Denoising language models (DLMs) trained with synthetic data achieve state-of-the-art ASR performance on Librispeech without external audio data, surpassing conventional language models and some self-supervised methods.
paper_ai_keywords: Denoising LM scaled error correction model text-to-speech systems ASR errors synthetic data multi-speaker TTS systems noise augmentation strategies Transformer-CTC ASR word error rate (WER) test-clean test-other Librispeech beam-search rescoring