Failing Forward: Improving Generative Error Correction for ASR with
  Synthetic Data and Retrieval Augmentation
Generative Error Correction (GEC) has emerged as a powerful post-processing
method to enhance the performance of Automatic Speech Recognition (ASR)
systems. However, we show that GEC models struggle to generalize beyond the
specific types of errors encountered during training, limiting their ability to
correct new, unseen errors at test time, particularly in out-of-domain (OOD)
scenarios. This phenomenon amplifies with named entities (NEs), where, in
addition to insufficient contextual information or knowledge about the NEs,
novel NEs keep emerging. To address these issues, we propose DARAG (Data- and
Retrieval-Augmented Generative Error Correction), a novel approach designed to
improve GEC for ASR in in-domain (ID) and OOD scenarios. We augment the GEC
training dataset with synthetic data generated by prompting LLMs and
text-to-speech models, thereby simulating additional errors from which the
model can learn. For OOD scenarios, we simulate test-time errors from new
domains similarly and in an unsupervised fashion. Additionally, to better
handle named entities, we introduce retrieval-augmented correction by
augmenting the input with entities retrieved from a database. Our approach is
simple, scalable, and both domain- and language-agnostic. We experiment on
multiple datasets and settings, showing that DARAG outperforms all our
baselines, achieving 8\% -- 30\% relative WER improvements in ID and 10\% --
33\% improvements in OOD settings.
6711bfbd035ddf7e14b22417
Sreyan Ghosh
claimed_verified
2024-10-28T15:36:39.302Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
user
Sreyan88
6711bfbd035ddf7e14b22418
Mohammad Sadegh Rasooli
admin_assigned
2024-10-18T12:58:14.910Z
658dd50b315340de5fcb1321
/avatars/72b0030ff3f5f47002e0ff4e2d665b46.svg
Mohammad Sadegh Rasooli
user
rasoolims
6711bfbd035ddf7e14b22419
Michael Levit
6711bfbd035ddf7e14b2241a
Peidong Wang
6711bfbd035ddf7e14b2241b
Jian Xue
6711bfbd035ddf7e14b2241c
Dinesh Manocha
admin_assigned
2024-10-18T12:59:11.285Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
6711bfbd035ddf7e14b2241d
Jinyu Li
Failing Forward: Improving Generative Error Correction for ASR with
  Synthetic Data and Retrieval Augmentation
Generative Error Correction (GEC) has emerged as a powerful post-processing
method to enhance the performance of Automatic Speech Recognition (ASR)
systems. However, we show that GEC models struggle to generalize beyond the
specific types of errors encountered during training, limiting their ability to
correct new, unseen errors at test time, particularly in out-of-domain (OOD)
scenarios. This phenomenon amplifies with named entities (NEs), where, in
addition to insufficient contextual information or knowledge about the NEs,
novel NEs keep emerging. To address these issues, we propose DARAG (Data- and
Retrieval-Augmented Generative Error Correction), a novel approach designed to
improve GEC for ASR in in-domain (ID) and OOD scenarios. We augment the GEC
training dataset with synthetic data generated by prompting LLMs and
text-to-speech models, thereby simulating additional errors from which the
model can learn. For OOD scenarios, we simulate test-time errors from new
domains similarly and in an unsupervised fashion. Additionally, to better
handle named entities, we introduce retrieval-augmented correction by
augmenting the input with entities retrieved from a database. Our approach is
simple, scalable, and both domain- and language-agnostic. We experiment on
multiple datasets and settings, showing that DARAG outperforms all our
baselines, achieving 8\% -- 30\% relative WER improvements in ID and 10\% --
33\% improvements in OOD settings.
DARAG enhances generative error correction for ASR by using synthetic data and retrieval-augmented methods to improve performance in both in-domain and out-of-domain scenarios.
Generative Error Correction (GEC) Automatic Speech Recognition (ASR) Named Entities (NEs) out-of-domain (OOD) in-domain (ID) LLMs text-to-speech models retrieval-augmented correction relative Word Error Rate (WER) improvements