title: Improving Joint Speech-Text Representations Without Alignment
summary: The last year has seen astonishing progress in text-prompted image generation
  premised on the idea of a cross-modal representation space in which the text
  and image domains are represented jointly. In ASR, this idea has found
  application as joint speech-text encoders that can scale to the capacities of
  very large parameter models by being trained on both unpaired speech and text.
  While these methods show promise, they have required special treatment of the
  sequence-length mismatch inherent in speech and text, either by up-sampling
  heuristics or an explicit alignment model. In this work, we offer evidence that
  joint speech-text encoders naturally achieve consistent representations across
  modalities by disregarding sequence length, and argue that consistency losses
  could forgive length differences and simply assume the best alignment. We show
  that such a loss improves downstream WER in both a large-parameter monolingual
  and multilingual system.
paper_authors._id: 64d9b13646bda7d8f53b0351
paper_authors.name: Cal Peyser
paper_authors._id: 64d9b13646bda7d8f53b0352
paper_authors.name: Zhong Meng
paper_authors._id: 64d9b13646bda7d8f53b0353
paper_authors.name: Ke Hu
paper_authors._id: 64d9b13646bda7d8f53b0354
paper_authors.name: Rohit Prabhavalkar
paper_authors._id: 64d9b13646bda7d8f53b0355
paper_authors.name: Andrew Rosenberg
paper_authors._id: 64d9b13646bda7d8f53b0356
paper_authors.name: Tara N. Sainath
paper_authors._id: 64d9b13646bda7d8f53b0357
paper_authors.name: Michael Picheny
paper_authors._id: 64d9b13646bda7d8f53b0358
paper_authors.name: Kyunghyun Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T00:52:02.244Z
paper_authors._id: 649c1b8485b05ee02723334a
paper_authors.avatarUrl: /avatars/29080f550fdb471c4d2daace86aff3b0.svg
paper_authors.fullname: Kyunghyun Cho
paper_authors.type: user
paper_authors.user: kyunghyuncho
paper_title: Improving Joint Speech-Text Representations Without Alignment
paper_summary: The last year has seen astonishing progress in text-prompted image generation
  premised on the idea of a cross-modal representation space in which the text
  and image domains are represented jointly. In ASR, this idea has found
  application as joint speech-text encoders that can scale to the capacities of
  very large parameter models by being trained on both unpaired speech and text.
  While these methods show promise, they have required special treatment of the
  sequence-length mismatch inherent in speech and text, either by up-sampling
  heuristics or an explicit alignment model. In this work, we offer evidence that
  joint speech-text encoders naturally achieve consistent representations across
  modalities by disregarding sequence length, and argue that consistency losses
  could forgive length differences and simply assume the best alignment. We show
  that such a loss improves downstream WER in both a large-parameter monolingual
  and multilingual system.
paper_ai_keywords: joint speech-text encoders very large parameter models unpaired speech and text sequence-length mismatch up-sampling heuristics explicit alignment model consistent representations modalities consistency losses best alignment WER large-parameter monolingual system multilingual system