title: Masked Audio Text Encoders are Effective Multi-Modal Rescorers
summary: Masked Language Models (MLMs) have proven to be effective for second-pass
  rescoring in Automatic Speech Recognition (ASR) systems. In this work, we
  propose Masked Audio Text Encoder (MATE), a multi-modal masked language model
  rescorer which incorporates acoustic representations into the input space of
  MLM. We adopt contrastive learning for effectively aligning the modalities by
  learning shared representations. We show that using a multi-modal rescorer is
  beneficial for domain generalization of the ASR system when target domain data
  is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and
  3%-7% on out-of-domain datasets, over the text-only baseline. Additionally,
  with very limited amount of training data (0.8 hours), MATE achieves a WER
  reduction of 8%-23% over the first-pass baseline.
paper_authors._id: 6462e5ec2538819c72a09a12
paper_authors.name: Jinglun Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:04:57.882Z
paper_authors._id: 6463c26aa5af935cfe70f08d
paper_authors.avatarUrl: /avatars/33b1210098891db54f57d1344b5110fb.svg
paper_authors.fullname: Jinglun (Jason) Cai
paper_authors.type: user
paper_authors.user: jasoncai
paper_authors._id: 6462e5ec2538819c72a09a13
paper_authors.name: Monica Sunkara
paper_authors._id: 6462e5ec2538819c72a09a14
paper_authors.name: Xilai Li
paper_authors._id: 6462e5ec2538819c72a09a15
paper_authors.name: Anshu Bhatia
paper_authors._id: 6462e5ec2538819c72a09a16
paper_authors.name: Xiao Pan
paper_authors._id: 6462e5ec2538819c72a09a17
paper_authors.name: Sravan Bodapati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:36:56.541Z
paper_authors._id: 640f6eb0a92fedb0e851c4ed
paper_authors.avatarUrl: /avatars/2d2e35ff7d064445ca68a79d71281fd1.svg
paper_authors.fullname: Sravan Bodapati
paper_authors.type: user
paper_authors.user: sravanbabu1991
paper_title: Masked Audio Text Encoders are Effective Multi-Modal Rescorers
paper_summary: Masked Language Models (MLMs) have proven to be effective for second-pass
  rescoring in Automatic Speech Recognition (ASR) systems. In this work, we
  propose Masked Audio Text Encoder (MATE), a multi-modal masked language model
  rescorer which incorporates acoustic representations into the input space of
  MLM. We adopt contrastive learning for effectively aligning the modalities by
  learning shared representations. We show that using a multi-modal rescorer is
  beneficial for domain generalization of the ASR system when target domain data
  is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and
  3%-7% on out-of-domain datasets, over the text-only baseline. Additionally,
  with very limited amount of training data (0.8 hours), MATE achieves a WER
  reduction of 8%-23% over the first-pass baseline.
paper_ai_keywords: Masked Language Models (MLMs) Masked Audio Text Encoder (MATE) multi-modal masked language model acoustic representations contrastive learning shared representations domain generalization word error rate (WER)