summaries: This paper is dedicated to proposing a self-supervised objective, local prior matching (LMP), for speech recognition. This approach can take advantage of vase quantities of unlabeled speech data. What' more, the objective is simple to implement and theoretically well-motivated. In the paper, based on a supervised pretrained model, it then finetunes 360 hours with unlabeled data and LPM reduces the WER consistently. They also conduct extensive ablation experiments to show the effect of their self-supervised approach. This work proposed a distillation approach which use ASRs to generate hypotheses for unsupervised data, run a LM to get probability for the hypothesis, and perform distillation with the resulting probability. The ASRs being used for generating hypotheses can be either a model trained with the supervised data or the student model, and can switch between the two during training. In the experiments, ASR models are pre-trained with the subset of Librispeech data and use the rest of Librispeech data as unsupervised data, and the LM is trained with Librispeech LM data. The experiments shown the proposed approach improve baseline model trained with the Librispeech subset significantly.
abstractText: We propose local prior matching (LPM), a self-supervised objective for speech recognition. The LPM objective leverages a strong language model to provide learning signal given unlabeled speech. Since LPM uses a language model, it can take advantage of vast quantities of both unpaired text and speech. The loss is theoretically well-motivated and simple to implement. More importantly, LPM is effective. Starting from a model trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data LPM reduces the WER by 26% and 31% relative on a clean and noisy test set, respectively. This bridges the gap by 54% and 73% WER on the two test sets relative to a fully supervised model on the same 360 hours with labels. By augmenting LPM with an additional 500 hours of noisy data, we further improve the WER on the noisy test set by 15% relative. Furthermore, we perform extensive ablative studies to show the importance of various configurations of our self-supervised approach.
references.authors: Jordi Adell Antonio Bonafonte
references.title: Towards phone segmentation for concatenative speech synthesis
references.venue: In Fifth ISCA Workshop on Speech Synthesis,
references.authors: Dario Amodei Sundaram Ananthanarayanan Rishita Anubhai Jingliang Bai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Qiang Cheng Guoliang Chen
references.title: Deep Speech 2: Endto-end speech recognition in English and Mandarin
references.venue: In Proc. International Conference on Machine Learning (ICML),
references.authors: Dzmitry Bahdanau Jan Chorowski Dmitriy Serdyuk Philemon Brakel Yoshua Bengio
references.title: Endto-end attention-based large vocabulary speech recognition
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Murali Karthick Baskar Shinji Watanabe Ramon Astudillo Takaaki Hori Lukáš Burget Jan Černockỳ
references.title: Semi-supervised sequence-to-sequence ASR using unpaired speech and text
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Yoshua Bengio Nicholas Léonard Aaron Courville
references.title: Estimating or propagating gradients through stochastic neurons for conditional computation
references.venue: arXiv preprint arXiv:1308.3432,
references.authors: Chung-Cheng Chiu Tara N Sainath Yonghui Wu Rohit Prabhavalkar Patrick Nguyen Zhifeng Chen Anjuli Kannan Ron J Weiss Kanishka Rao Ekaterina Gonina
references.title: State-of-the-art speech recognition with sequence-to-sequence models
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Noam Chomsky
references.title: Knowledge of language: Its nature, origin, and use
references.authors: Jan Chorowski Navdeep Jaitly
references.title: Towards better decoding and language model integration in sequence to sequence models
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Jia Cui Brian Kingsbury Bhuvana Ramabhadran George Saon Tom Sercu Kartik Audhkhasi Abhinav Sethy Markus Nussbaum-Thom Andrew Rosenberg
references.title: Knowledge distillation across ensembles of multilingual models for low-resource languages
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Yann N Dauphin Angela Fan Michael Auli David Grangier
references.title: Language modeling with gated convolutional networks
references.venue: In Proc. International Conference on Machine Learning (ICML),
references.authors: Jennifer Drexler James Glass
references.title: Combining end-to-end and adversarial training for low-resource speech recognition
references.venue: In Proc. IEEE Spoken Language Technology Workshop (SLT),
references.authors: Emmanuel Dupoux
references.title: Cognitive science in the era of artificial intelligence: A roadmap for reverseengineering the infant
references.venue: language-learner. Cognition,
references.authors: James Glass
references.title: Towards unsupervised speech processing
references.venue: In International Conference on Information Science, Signal Processing and their Applications (ISSPA),
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Proc. Neural Information Processing Systems (NeurIPS),
references.authors: Awni Hannun Carl Case Jared Casper Bryan Catanzaro Greg Diamos Erich Elsen Ryan Prenger Sanjeev Satheesh Shubho Sengupta Adam Coates
references.title: Deep Speech: Scaling up end-to-end speech recognition
references.venue: arXiv preprint arXiv:1412.5567,
references.authors: Awni Hannun Ann Lee Qiantong Xu Ronan Collobert
references.title: Sequence-to-sequence speech recognition with time-depth separable convolutions
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Tomoki Hayashi Shinji Watanabe Yu Zhang Tomoki Toda Takaaki Hori Ramon Astudillo Kazuya Takeda
references.title: Back-translation-style data augmentation for end-to-end ASR
references.venue: In Proc. IEEE Spoken Language Technology Workshop (SLT),
references.authors: Takaaki Hori Ramon Astudillo Tomoki Hayashi Yu Zhang Shinji Watanabe Jonathan Le Roux
references.title: Cycle-consistency training for end-to-end speech recognition
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Wei-Ning Hsu James Glass
references.title: Extracting domain invariant features by unsupervised learning for robust automatic speech recognition
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Wei-Ning Hsu Yu Zhang James Glass
references.title: Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation
references.venue: In Proc. IEEE Workshop on Automfatic Speech Recognition and Understanding (ASRU),
references.authors: Wei-Ning Hsu Hao Tang James Glass
references.title: Unsupervised adaptation with interpretable disentangled representations for distant conversational speech recognition
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Eric Jang Shixiang Gu Ben Poole
references.title: Categorical reparameterization with gumbel-softmax
references.venue: In Proc. International Conference on Learning Representations (ICLR),
references.authors: Jacob Kahn Ann Lee Awni Hannun
references.title: Self-training for end-to-end speech recognition
references.venue: arXiv preprint arXiv:1606.07947,
references.authors: Shigeki Karita Shinji Watanabe Tomoharu Iwata Atsunori Ogawa Marc Delcroix
references.title: Semisupervised end-to-end speech recognition
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Taku Kudo
references.title: Subword regularization: Improving neural network translation models with multiple subword candidates
references.venue: In Proc. Annual Meeting of the Association for Computational Linguistics (ACL),
references.authors: Taku Kudo John Richardson
references.title: Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
references.venue: In Proc. Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Patricia K. Kuhl
references.title: Early language acquisition: cracking the speech code
references.venue: Nature Reviews Neuroscience,
references.authors: Bo Li Tara N Sainath Arun Narayanan Joe Caroselli Michiel Bacchiani Ananya Misra Izhak Shafran Hasim Sak Golan Pundak Kean K Chin
references.title: Acoustic modeling for Google home
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Bo Li Tara N Sainath Ruoming Pang Zelin Wu
references.title: Semi-supervised training for end-to-end models via weak distillation
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Alexander H Liu Hung-yi Lee Lin-shan Lee
references.title: Adversarial training of end-to-end speech recognition using a criticizing language model
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Vimal Manohar Hossein Hadian Daniel Povey Sanjeev Khudanpur
references.title: Semi-supervised training of acoustic models using lattice-free MMI
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Zhong Meng Zhuo Chen Vadim Mazalov Jinyu Li Yifan Gong
references.title: Unsupervised adaptation with domain separation networks for robust speech recognition
references.venue: In Proc. IEEE Workshop on Automfatic Speech Recognition and Understanding (ASRU),
references.authors: Zhong Meng Jinyu Li Yifan Gong Biing-Hwang Juang
references.title: Adversarial teacher-student learning for unsupervised domain adaptation
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Zhong Meng Jinyu Li Yifan Gong
references.title: Attentive adversarial learning for domain-invariant training
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Ramesh Nallapati Bowen Zhou Caglar Gulcehre Bing Xiang
references.title: Abstractive text summarization using sequence-to-sequence RNNs and beyond
references.venue: In Proc. ACL Conference on Natural Language Learning (CoNLL),
references.authors: Vassil Panayotov Guoguo Chen Daniel Povey Sanjeev Khudanpur
references.title: Librispeech: an ASR corpus based on public domain audio books
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Sree Hari Krishnan Parthasarathi Nikko Strom
references.title: Lessons from building acoustic models with a million hours of speech
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Kainan Peng Wei Ping Zhao Song Kexin Zhao
references.title: Parallel neural text-to-speech
references.venue: arXiv preprint arXiv:1905.08459,
references.authors: Vineel Pratap Awni Hannun Qiantong Xu Jeff Cai Jacob Kahn Gabriel Synnaeve Vitaliy Liptchinsky Ronan Collobert
references.title: wav2letter++: The fastest open-source speech recognition system
references.venue: arXiv preprint arXiv:1812.07625,
references.authors: Kristen Precoda
references.title: Non-mainstream languages and speech recognition: Some challenges
references.venue: CALICO journal,
references.authors: Odette Scharenborg Vincent Wan Mirjam Ernestus
references.title: Unsupervised speech segmentation: An analysis of the hypothesized phone boundaries
references.venue: The Journal of the Acoustical Society of America,
references.authors: Rico Sennrich Barry Haddow Alexandra Birch
references.title: Improving neural machine translation models with monolingual data
references.venue: In Proc. Annual Meeting of the Association for Computational Linguistics (ACL),
references.authors: Nitish Srivastava Geoffrey Hinton Alex Krizhevsky Ilya Sutskever Ruslan Salakhutdinov
references.title: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning
references.authors: Sining Sun Binbin Zhang Lei Xie Yanning Zhang
references.title: An unsupervised deep domain adaptation approach for robust speech
references.venue: recognition. Neurocomputing,
references.authors: Andros Tjandra Sakriani Sakti Satoshi Nakamura
references.title: Listening while speaking: Speech chain by deep learning
references.venue: In Proc. IEEE Workshop on Automfatic Speech Recognition and Understanding (ASRU),
references.authors: Andros Tjandra Sakriani Sakti Satoshi Nakamura
references.title: End-to-end feedback loss in speech chain framework via straight-through estimator
references.venue: In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Proc. Neural Information Processing Systems (NeurIPS),
references.authors: Karel Veselỳ Lukás Burget Jan Cernockỳ
references.title: Semi-supervised DNN training with word selection for ASR
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Yu-Hsuan Wang Cheng-Tao Chung Hung-yi Lee
references.title: Gate activation signal analysis for gated recurrent neural networks and its correlation with phoneme boundaries
references.venue: In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH),
references.authors: Ronald J Williams
references.title: Simple statistical gradient-following algorithms for connectionist reinforcement learning
references.venue: Machine learning,
references.authors: Zichao Yang Zhiting Hu Chris Dyer Eric P Xing Taylor Berg-Kirkpatrick
references.title: Unsupervised text style transfer using language models as discriminators
references.venue: In Proc. Neural Information Processing Systems (NeurIPS),
references.authors: Neil Zeghidour Qiantong Xu Vitaliy Liptchinsky Nicolas Usunier Gabriel Synnaeve Ronan Collobert
references.title: Fully convolutional speech recognition
references.venue: arXiv preprint arXiv:1812.06864,
references.authors: Jun-Yan Zhu Taesung Park Phillip Isola Alexei A Efros
references.title: Unpaired image-to-image translation using cycle-consistent adversarial networks
references.venue: In Proc. IEEE International Conference on Computer Vision (ICCV),
sections.heading: 1 INTRODUCTION
sections.text: Fully supervised learning remains the mainstream paradigm for state-of-the-art automatic speech recognition (ASR). These systems require huge annotated data sets (Li et al., 2017; Chiu et al., 2018; Hannun et al., 2014; Amodei et al., 2016), which are time-consuming and expensive to collect. This hinders the development of accurate ASR systems for low resource languages (Precoda, 2013). In fact, out of over 6,000 spoken languages, fewer than 150 are supported by commercial ASR service providers. In sharp contrast to how we teach machines to recognize speech, humans do not learn by listening to thousand hours of speech and simultaneously reading the corresponding transcriptions. Instead, as noted in (Chomsky, 1986; Kuhl, 2004; Glass, 2012; Dupoux, 2018), humans possess an inherent ability to learn from vast quantities of unlabeled speech. Consider the case of conversing with someone with a strong accent. Even when the speaker pronounces several words in an unusual way, one can often correctly understand the sentence. We argue that the source of indirect supervision in processing unlabeled speech comes from prior knowledge about the world and the context of the speech.
  Inspired by this, we devise a self-supervised learning framework termed local prior matching (LPM). We apply this framework to speech recognition allowing an ASR model to learn from unlabeled speech by leveraging a strong language model, which serves as the prior for self-supervision. Given an unlabeled utterance, the ASR model proposes multiple hypotheses and the language model provides a learning signal by evaluating the plausibility of each one.
  We evaluate the LPM method on the LibriSpeech corpus (Panayotov et al., 2015), using 100 hours of labeled speech to seed the proposal model. Using 360 hours of additional labeled data reduces the word error rate (WER) by 3.8% and 13.1% absolute on an easier and a more-challenging test set, respectively. Using the same 360 hours but without labels, LPM reduces the WER by 2.1% and 9.5% absolute on the same two test sets, effectively bridging the gap to fully-supervised learning by 54% and 73%. In addition, by augmenting LPM with another 500 hours, for a total of 860 hours of unlabeled speech, LPM reduces WER on the noisier test set by 14.2% in total. Hence, LPM is able to surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data. We also conduct extensive ablation studies and analyses in order to demonstrate the significance of each proposed component. For reproducibility, software will be open source and made publicly available in the camera-ready version.
sections.heading: 2 METHOD
sections.heading: 2.1 PRELIMINARIES
sections.text: Let x denote an utterance of speech and y be a transcription. We assume speech is generated following a two-step process: y ∼ py x ∼ px|y, (1) where the text y is first generated from the language model (LM) py, and the speech x is then generated from a text-to-speech (TTS) model px|y conditioned on y. The posterior py|x is then the ASR model of interest. For convenience, we refer to pxy(x,y) = py(y) px|y(x | y) as the joint distribution, and px(x) = ∑ y pxy(x,y) as the marginal distribution. In a typical supervised learning setting, one has access to a labeled dataset Dl, which contains paired samples {(x(i),y(i))}Ni=1 drawn from the joint distribution. An ASR model qy|x can be trained by minimizing the marginal-weighted KL-divergence: Ex∼px [DKL(py|x || qy|x)], which can be estimated with∑
  (x,y)∈Dl − log qy|x(y | x) + const from samples (i.e. the commonly used cross-entropy loss). In the self-supervised setting we only have access to unlabeled speech dataset Dsu, yet we still wish to optimize the same marginal weighted KL-divergence. Since we do not have samples from py|x=x directly, we can, under mild assumptions, estimate it.
  Assumption 1 Different y can lead to the same pronunciation. One reason is the tokenization of words may not be unique. For example y can be the sequence of word-pieces “c at ” or “ca t ”, both of which result in the same spoken word. Homophones (e.g. “right” and “write”) also have the same pronunciation. To formalize this property, let f : Y → Z be a function which maps a text sequence to its pronunciation z ∈ Z (a broader definition of “pronunciation” here, where z = f(y) is a distribution over phoneme sequences for a given text y), and let px|y(x | y) = px|z(x | f(y)). We let the set Yz = {y : f(y) = z} ⊂ Y contain all text samples with the same pronunciation and hence the same conditional distribution, px|z=z .
  Assumption 2 We assume that two pairs (y,y′) that have different pronunciations (i.e., f(y) 6= f(y′)) either sound similar or are very different. For instance, “lose” and “loose” result in almost the same pronunciation whereas an English speaker would rarely mistake “winter is coming” for “you know nothing”. In the former case, the conditional distributions px|y(x | y) and px|y(x | y′) may overlap substantially whereas in the latter case they barely intersect. More formally, for each z, we assume the pronunciation space Z can be partitioned by acoustic similarity asZ+z ∪Z−z . For all z+ ∈ Z+z , the conditional distributions are similar to that of z, such that DKL(px|z=z || px|z=z+) ≤ for small . On the other hand, we assume the set of dissimilar pronunciations Z−z contributes very little to the marginal probability of non-outlier speech samples. In other words, for any z and x where px|z(x | z) ≥ δ, we assume ∑ z−∈Z−z ∑ y−∈Yz− px|y(x | y−) py(y−) px(x).
sections.heading: 2.2 LOCAL PRIOR MATCHING
sections.text: Equipped with the above assumptions, we next present a tractable approximation of the posterior py|x using unlabeled speech data. Figure 1 shows a toy example of the proposed approach. Let x be a non-outlier speech sample that is generated conditioned on a text sample y∗, whose pronunciation is f(y∗) = z∗. The posterior can be derived with Bayes’ rule as follows:
  py|x(y | x) = py(y) px|y(x | y)∑
  ŷ∈Y py(ŷ) px|y(x | ŷ) = py(y) px|y(x | y)∑ ẑ∈Z+
  z∗∪Z − z∗
  ∑ ŷ∈Yẑ py(ŷ) px|y(x | ŷ) (2)
  ≈ py(y) px|y(x | y)∑
  ẑ∈Z+ z∗
  ∑ ŷ∈Yẑ py(ŷ) px|y(x | ŷ) · 1∪ ẑ∈Z+ z∗ Yẑ (y) (3)
  = py(y) px|z(x | f(y))∑
  ẑ∈Z+ z∗
  ∑ ŷ∈Yẑ py(ŷ) px|z(x | ẑ) · 1∪ ẑ∈Z+ z∗ Yẑ (y). (4)
  The set Y can be partitioned according to pronunciation to arrive at equation 2, which is then approximated using by remark 2 to get equation 3, where 1 is the indicator function. In equation 4, px|y(x | ŷ) is simply replaced with px|z(x | f(ŷ)) as stated in remark 1. Despite the simplification, estimating equation 4 requires a language model, a TTS model, and the enumeration of text sequences with similar pronunciations to that of z∗. To avoid the latter two requirements, we make the following approximations:
  1. Replace px|z(x | ẑ) with px|z(x | z∗) for all ẑ ∈ Z+z∗ by exploiting the assumption that these distributions are very similar to each other (i.e., from remark 2 we have DKL(px|z=z∗ || px|z=ẑ) ≤ ).
  2. Replace ∪ẑ∈Z+ z∗ Yẑ with hypotheses proposed from a beam search using a bootstrapped
  ASR proposal model. We refer to the proposal model as ry|x and the set of hypotheses for x generated from the beam search as B(ry|x,x, k), where k is the beam size.
  The approximated posterior p̄y|x(y | x) can then be written as:
  p̄y|x(y | x) = py(y) px|z(x | z∗)∑
  ŷ∈B(ry|x,x,k) py(ŷ) px|z(x | z ∗) · 1B(ry|x,x,k)(y) (5)
  = py(y)∑
  ŷ∈B(ry|x,x,k) py(ŷ) · 1B(ry|x,x,k)(y). (6)
  The approximated posterior only requires computing language model probabilities of the beam search hypotheses, which is tractable. We refer to equation 6 as the local prior, since it is a renormalized distribution in the neighborhood of y∗. We also propose local prior matching (LPM) as an unsupervised objective for training an ASR model qy|x: Llpm(qy|x;x, py, ry|x, k) = DKL(p̄y|x=x||qy|x=x)
  = −H(p̄y|x=x)− ∑
  y∈B(ry|x,x,k)
  py(y)∑ ŷ∈B(ry|x,x,k) py(ŷ) log qy|x(y | x),
  which minimizes the KL-divergence between the local prior and the model distribution. For clarity, we term qy|x the online model, and let qy|x(y | x; θq) and ry|x(y | x; θr) denote the models that are parameterized by and θq and θr, respectively. Since the LPM objective simply re-weights hypotheses by the local prior, we do not need a continuous relaxation of y to make py differentiable with respect to θq as in Yang et al. (2018); Liu et al. (2019). The gradient of the proposed LPM objective is given by:
  ∂Llpm(qy|x;x, py, ry|x, k) ∂θq
  = ∑
  y∈B(ry|x,x,k)
  py(y)∑ ŷ∈B(ry|x,x,k) py(ŷ) ∂ log qy|x(y | x) ∂θq .
  We use paired audio and text to estimate ry|x and unpaired speech to compute the LPM objective. The LPM objective can also take advantage of large unpaired text corpora in order to obtain better language models which directly improve the quality of the approximate posterior.
sections.heading: 2.3 PROPOSAL MODEL
sections.text: The quality of the posterior approximation p̄y|x depends on the proposal model ry|x. Instead of using a fixed proposal model throughout the entire training process, we update ry|x with qy|x if the latter is better.
  On-policy beam search The first approach always uses the online model qy|x as the proposal model. This means θr = θq and is effectively a form of on-policy beam search, since the model used to generate hypotheses is also the model we update.
  Off-policy beam search While the on-policy method benefits from the immediate improvement of the online model, it also suffers immediately if the gradient update from a mini-batch deteriorates performance. This can result in instability during optimization. We consider a second option which does not tie θr and θq but instead updates θr with θq every T steps only when the performance of the online model qy|x is better than that of the proposal model ry|x by some metric. We refer to the second option as off-policy beam search. To avoid overfitting to the training set, we use the character error rate (CER) on the validation set as the metric for the proposal model update. We set T = 1000 for all experiments with the off-policy beam search.
sections.heading: 2.4 FILTERING HYPOTHESES USING ESTIMATED LENGTHS
sections.text: As noted in Chorowski & Jaitly (2017), sequence-to-sequence ASR models sometimes predict endof-sentence (EOS) tokens too early or generate looping n-grams, resulting in hypotheses that are significantly shorter or longer than the set of acoustically matched texts for a given utterance. Of the two failure modes, the former is more harmful to training with LPM. The reason is that the LPM objective assumes all hypotheses obtained from beam search are acoustically reasonable, and weights each of them by linguistic plausibility given by an LM. While LMs are effective in discriminating plausibility between sentences of similar lengths, we find empirically they tend to assign higher probabilities to shorter sentences than to longer sentences, even when the longer ones are more plausible and grammatically correct than the shorter ones. As a result, truncated hypotheses are assigned higher weights than those that are acoustically matched but longer, which in turn encourages earlier prediction of EOS tokens and forms a catastrophic feedback loop particularly with the on-policy beam search.
  To address this issue, we propose a simple filtering heuristic based on the text length. Before training the model, a text length L is estimated for each unlabeled speech sample x. During training, only hypotheses with length close to L are retained for the LPM objective computation. Let len(y) denote the length of y. We keep a hypothesis y only if brlb · Lc ≤ len(y) ≤ drub · Le, where rlb and rub are the text length lower and upper bound ratios, respectively. Several methods can be used to estimate the text length on an unlabeled utterance, including using the average speaking rate (Peng et al., 2019) or generating a phoneme/syllable segmentation (Adell & Bonafonte, 2004; Scharenborg et al., 2010; Wang et al., 2017). In this work, we estimate the length by using using that of the best hypothesis generated from the initial proposal model, generated with either ASR-only greedy decoding or ASR+LM beam search decoding.
sections.heading: 3 EXPERIMENTAL SETUP
sections.text: Dataset We evaluate our approach on LibriSpeech (Panayotov et al., 2015), a crowd-sourced audio book corpus derived from the LibriVox Project. The training set contains 960 hours of speech, officially split into three sets: train-clean-100, train-clean-360, and train-other-500, where the first two sets are easier and the third set is noisier and more accented. Similarly, the development and test sets are also split according to difficulty, resulting in four partitions: {dev, test}×{clean, other}, each of which contains roughly five hours of speech. In this work we use train-clean-100 as the paired speech data, and the other two training splits as the unpaired data.
  We train the language model on the unpaired text data provided with LibriSpeech, which includes approximately 14,500 books collected from Project Gutenberg. Some of the books in the text corpus overlap with those in the LibriSpeech training set. To avoid training the LM on the ground truth text of the unlabeled speech, we exclude the 997 overlapping books from the text data. We follow the same recipe as Kahn et al. (2019) to pre-process the remaining text.
  Neural Network Architecture The proposal model ry|x and the online model qy|x are sequenceto-sequence neural networks (Bahdanau et al., 2016; Chorowski & Jaitly, 2017) with the same timedepth separable (TDS) architecture proposed in Hannun et al. (2019). The encoder is fully convolutional, composed of TDS blocks which reduce the number of parameters while keeping the receptive field large. The decoder is a single layer recurrent neural network (RNN) with gated recurrent units (GRUs), equipped with a single-headed inner-product key-value attention (Vaswani et al., 2017) for querying information from the encoder outputs. We follow the recipe of Kahn et al. (2019) which uses fewer TDS blocks in the encoder compared to Hannun et al. (2019) in order to generalize better when trained on the smaller LibriSpeech train-clean-100. The output target of the decoder at each step is a posterior distribution over 5,000 word pieces, generated with the SentencePiece toolkit (Kudo & Richardson, 2018) using transcripts from train-clean-100.
  To enable efficient evaluation of the language model probabilities, which is required at each training step, we use the gated convolutional language model architecture (ConvLM) proposed in Dauphin et al. (2017), which achieves competitive performances compared to recurrent models while significantly reducing the latency. We use the same 5,000 word-piece vocabulary for the LM which is trained with the same model configuration and recipe as Zeghidour et al. (2018). The trained ConvLM achieves a token perplexity of 34.24 on the development set.
  Optimization We use both paired and unpaired data to optimize qy|x. To simplify the optimization procedure, the model is provided with either a paired or an unpaired batch at each step, alternated with a fixed ratio ml : mu. When given a paired batch of n samples, {(x(i),y(i))}ni=1, the model minimizes the standard cross-entropy loss, 1n ∑ i− log qy|x(y(i) | x(i)). When provided with an unpaired batch {x(i)}ni=1, the model minimizes a weighted LPM loss, αn ·∑ i Llpm(qy|x;x(i), py, ry|x, k). The weight α and the mixing ratio ml/mu are used to balance the supervised and self-training objectives. For regularization we use 20% dropout (Srivastava et al., 2014), 10% label smoothing, 1% decoder input sampling, and 1% word piece sampling (Kudo, 2018) following Kahn et al. (2019). We use SGD without momentum to train the online model with an initial learning rate of 5e-2. To achieve a good CER on the development sets, the model is trained for at least 1.6M steps (paired and unpaired ones combined) with a batch size of 16 (8 GPUs × 2 per GPU). The learning rate is annealed by a factor of two every 0.64M steps. All experiments in this paper are implemented in the wav2letter++ framework (Pratap et al., 2018).
  Initialization To initialize the proposal model and the online model, we consider three checkpoints from a baseline model trained on train-clean-100 for a varying number of steps using only the supervised objective. The three checkpoints, whose parameters are denoted as θA, θB , and θC , are trained for about 300k / 40k / 16k steps, achieving average development set CERs of 13% / 20% / 38%, respectively. To obtain a better approximation p̄y|x for the posterior, it is desirable to use the best-performed checkpoint to initialize the proposal model. By contrast, as observed in Kahn et al. (2019) that training from scratch achieves consistently better performances than starting from a well-trained model, we hypothesize that initializing the online model with an earlier checkpoint may also have a similar effect. Hence, we initialize θr = θA and θq = θC if not otherwise specified.
sections.heading: 4 RESULTS
sections.text: The best supervised model trained only on train-clean-100 (θA) achieves a WER of 14.00%/37.02% on dev-clean/dev-other, respectively. Unless otherwise stated, we use train-clean-360 as the unpaired speech dataset, (rlb, rub) = (0.95, 1.05) for length filtering with reference lengths obtained from ASR-only greedy decoding.
sections.heading: 4.1 BEAM SIZE, MIXING RATIO, AND LPM WEIGHTS
sections.text: Table 1 shows how the WER varies with the beam size and the mixing ratio. for all mixing ratios, the model improves the most from beam size of k = 1 to k = 2, showing the benefit of considering multiple hypotheses. The improvement is greater when a higher mixing ratio of unpaired-to-paired speech is used. In addition, we note that the LM is effectively unused when k = 1 because the LM probability assigned to each hypothesis is normalized within the beam. If there is only one hypothesis, it will be assigned an approximate posterior probability of one. The amount of improvement
  diminishes with larger beam sizes, and the performance even starts to degrade beyond k = 4 when using a higher mixing ratio. This may result from the inclusion of worse hypotheses which have a better scored under the LM. We use the best setting for the following experiments with a mixing ratio rl : ru = 1 : 4, a beam size k = 4, and an LPM weight α = 0.2. We present detailed results varying the LPM weight in the Appendix.
sections.heading: 4.2 PROPOSAL MODEL UPDATE AND MODEL INITIALIZATION
sections.text: In addition to the two proposal model update strategies proposed in Section 2.3, termed on-policy and off-policy (better), we experiment with two additional strategies. The first, off-policy (never), uses a fixed proposal model throughout training without ever updating. The second, off-policy (always), updates the proposal model with the online model every T steps (i.e., set θr ← θq) regardless of the performance.
  The full results are shown in Table 2. Four key takeaways are as follows. (1) For all combinations of (ry|x, qy|x) initialization, off-policy (never) is the worst. This demonstrates the importance of updating the proposal model to generate better hypotheses during training. (2) Off-policy (always) consistently outperforms on-policy. We observe that training is significantly stabilized by reducing the proposal model update frequency from every step to every 1,000 steps. The effect is particularly prominent when initializing ry|x and qy|x from an earlier checkpoint (9.62% vs 20.02% on dev-clean, and 27.51% vs 45.62% on dev-other). (3) Off-policy (better) achieves the best WER in all settings and outperforms off-policy (always) by a larger margin when initializing from an earlier checkpoint. (4) Unlike the other strategies, off-policy (better) demonstrates consistent improvement when using a less-trained initial online model. In the following experiments, we initialize models with θr = θA and θq = θC unless otherwise specified.
sections.heading: 4.3 LENGTH FILTERING
sections.text: Table 3 shows the impact of length filtering with different proposal model update strategies, where both the proposal model and the online model are initialized with θA. As discussed in Section 2.4, on-policy suffers more than off-policy without length filtering. If the proposal model is never updated, then length filtering does not affect the final WER. We hypothesize that length filtering keeps the proposal model stable during training. Table 4 compares three different reference lengths: the oracle length (Oracle), the predicted length from ASR + LM beam search decoding (ASR + LM Dec), and the predicted length from ASR-only greedy decoding (ASR-only Dec). We observe that the WER does not differ much when using different reference lengths for filtering.
  Table 4: Length filtering using different reference lengths.
  Reference Length L dev-{clean / other} Oracle 8.85 / 26.39 ASR + LM Dec 8.99 / 26.36 ASR-only Dec 9.00 / 26.47
sections.heading: 4.4 CHOICE OF LANGUAGE MODELS
sections.text: We study how the LM affects the performance by experimenting with three alternative hypothesis weighting methods. The first method uses the LM score normalized by token length, py(y)1/len(y), which avoids favoring overly short hypotheses. The second method computes the LM probability of each hypothesis and shuffles them, which preserves the entropy of the original approximate posterior but randomizes the weights. The third assigns an equal weight for each hypothesis. Table 5 presents the results with three beam sizes: k = 2, 4, 8. Using the LM weights directly performs the best for all beam sizes. Length normalization (LenNorm) is better than shuffled and uniform, which have similar performance. This suggests that both ranking hypotheses correctly and scaling the weights proportional to the LM probability are important.
  We also observe how the quality of the LM affects the results. We test LMs of various quality by using the same ConvLM but trained for a different number of steps. We quantify LM quality by the token perplexity (PPL) on the development set. Table 6 shows a clear positive correlation between the LM quality and the final WER. This is expected given that the better LM results in a more accurate posterior approximation p̄y|x.
sections.heading: 4.5 FINAL RESULTS AND COMPARISON WITH LITERATURE
sections.text: The best performing model is trained for 3.2M steps, with a learning rate annealed by a factor of two every 1.28M steps when using 360 hours of unpaired speech, and every 0.64M steps when using 860 hours of unpaired speech. Reference lengths for filtering are obtained from ASR+LM beam search decoding. We compare LPM to fully supervised models and a pseudo labeling (PL) method in Table 7. Compared to LPM, PL demands extensive hyper-parameter tuning for the ASR+LM beam search as well as heuristic data filtering methods. to obtain high quality pseudo transcripts. We
  chose PL as a baseline because we follow the same experimental setup, and, more importantly, it achieves the state-of-the-art results on LibriSpeech when using train-clean-100 as paired data and train-other-360 as unpaired speech. A more complete table with results from the literature is in the Appendix.
  The upper half of Table 7 shows greedy decoding results without an LM. For the fully supervised model, when removing train-clean-360 the WER increases by 6.86% on test-clean and 13.36% on test-other. Using train-clean-360 speech without transcripts, LPM reduces the absolute WER by 5.64% and 12.21% on the two test sets, which recovers 82% and 91%, respectively, of the WER drop from removing the labels. Adding noisier train-other-500 to the unpaired set (total 860hr Dsu) further reduces the WER, and LPM achieves a better WER on the noisy sets (dev-other and testother) compared to the supervised model trained on 460 hours of clean paired data. In addition, LPM outperforms PL in all settings. This trend is consistent even when decoding with a strong ConvLM.
sections.heading: 5 ANALYSIS
sections.heading: 5.1 HYPOTHESIS QUALITY OF UNLABELED TRAINING SPEECH
sections.text: As discussed in Section 4.2, updating the proposal model is crucial to improve the hypotheses used during training. To quantify the improvement, Table 8 shows WERs on the unlabeled data of an LPM model at the beginning (step=0) and at the end (step=3.2M) of training. Note that this is a proxy of quality for LPM, since multiple hypotheses are used when setting k ≥ 2. We compare to the WER of the pseudo transcripts used in PL. Although generating hypotheses without an LM leads to worse WER than that of PL, toward the end of training, the proposal model of LPM produces better predictions on both train-clean-360 and train-other-500 than the fixed ones used in PL (ASR+LM stable). Furthermore, the WER on train-other-500 much higher for PL (21.51%) than for LPM at the end of training (13.00%), which explains why LPM achieves much better performances than PL when using the full 860hr of unpaired data.
sections.heading: 5.2 LINGUISTIC PLAUSIBILITY
sections.text: We expect models trained with LPM to generate more semantically and grammatically correct text since the ASR model receives direct supervision from the LM. Table 9 shows the proposed hypotheses for two utterances using a supervised baseline model and a model trained with LPM. The baseline model proposes erroneous hypotheses which are easy to discard even without the audio or the ground truth transcript. On the other hand, LPM generates hypotheses that are both grammatically and semantically plausible, with acceptable substitution errors in some cases (e.g., might/would).
  We also notice in Table 9 that the LM probabilities correlate well with linguistic plausibility for texts of similar lengths (e.g., hypotheses of the same utterance), but not for texts with large differences in length. Motivated by this observation, we propose to quantify linguistic plausibility of an ASR model by measuring the LM token perplexity exp( ∑ y log py(y)/ ∑ y len(y)) of the hypotheses on the development set obtained using ASR-only greedy decoding. Results are shown in Table 10. The ground truth text has the lowest perplexity on both sets as expected. We also observe that while all models are worse on dev-other than on dev-clean, LPM exhibits the smallest perplexity difference between the two sets, demonstrating how it successfully distills knowledge from the LM.
sections.heading: 6 RELATED WORK
sections.text: Our work builds on a large body of work in semi-supervised learning for ASR. Research in this direction can be classified based on the required modules and objectives used with unpaired data.
  In Drexler & Glass (2018) and Karita et al. (2018), bi-encoder network architectures are used, which map text and speech to representations in a shared space with their corresponding encoder, and then apply a shared decoder to map from the shared space to the text space. Another line of work adds a TTS model (Tjandra et al., 2017; 2019; Baskar et al., 2019) or a text-to-encoding (TTE) model (Hayashi et al., 2018; Hori et al., 2019) in the loop of ASR training, which can be utilized for back-translation style data augmentation (Sennrich et al., 2016) or cycle-consistency training (Zhu et al., 2017). Liu et al. (2019) treats ASR as a generative model that conditions on speech instead of random noise vectors, and adopts the generative adversarial network (GAN) (Goodfellow et al., 2014) framework in order to improve the fidelity of ASR-generated texts (Liu et al., 2019). All the aforementioned methods involve additional modules that must be jointly optimized with the ASR model and require finding a careful balance between multiple training objectives. Moreover, some methods require gradient to flow through samples drawn from the ASR outputs, and hence rely on techniques like Gumbel-softmax (Jang et al., 2017), straight-through estimator (Bengio et al., 2013), or REINFORCE (Williams, 1992). The LPM objective can evaluate output quality yet is fully differentiable.
  Knowledge distillation is also easy to used for semi-supervised learning (Cui et al., 2017; Li et al., 2019; Parthasarathi & Strom, 2019) because the teacher model can provide a training target for the unlabeled speech. Rather than learning from a teacher ASR model, LPM learns from a teacher LM.
  Pseudo labelling (Veselỳ et al., 2017; Manohar et al., 2018; Kahn et al., 2019) is most similar to our approach. Pseudo labelling uses a seed ASR model to transcribe unlabeled speech. The ASR model is then trained on the predicted labels along with the original labeled data used to train the seed model. In addition to being well motivated theoretically, we highlight the key methodological advantages of LPM below. (1) While previous work uses fixed pseudo transcripts throughout training and focuses on data filtering, we demonstrate the benefit of generating hypotheses on-the-fly as the proposal model improves. This is efficient since we only use an end-to-end ASR model for beam search decoding. (2) We motivate and show empirically the benefit from considering multiple hypotheses rather than just the top hypothesis (Veselỳ et al., 2017; Kahn et al., 2019). (3) Manohar et al. (2018) consider multiple hypotheses in a numerator lattice and maximize the aggregated probability of the lattice regardless as to how the probability is allocated amongst the word sequences. In contrast, we verify in the LM ablation study that considering the allocation of the posterior probability amongst the hypotheses is important.
  Aside from semi-supervised learning, this work is also related to unsupervised domain adaptation for ASR, where unlabeled speech of the target domain is provided. Unlike the proposed method, previous studies mainly focus on learning domain invariant features (Sun et al., 2017; Hsu & Glass, 2018; Meng et al., 2017; 2018; 2019) or data augmentation through learned speech transformation (Hsu et al., 2017; 2018).
sections.heading: 7 CONCLUSION AND FUTURE WORK
sections.text: We introduce local prior matching, a self-supervised learning objective for speech recognition, and demonstrate note-able reductions in WER when with the addition of unpaired audio and text. We also perform an extensive empirical study to demonstrate the importance of various configurations of LPM. While LPM is motivated by how humans learn to recognize speech, the proposed method can be applied to other sequence transduction tasks including machine translation (Sennrich et al., 2016) and text summarization (Nallapati et al., 2016), provided a good prior for the domain.
  We consider two promising directions for future in self-supervised speech recognition with LPM and in general: (1) Empower the model with more capabilities, such as learning to select whether or not to use a sample for self-supervision. (2) Using more context signals for prior estimation which can consider previous sentences as well as inputs from other modalities.
sections.heading: A MORE COMPREHENSIVE COMPARISON WITH THE LITERATURE
sections.heading: C CHARACTER ERROR RATES
sections.heading: B ADDITIONAL RESULTS