summaries: This paper conducts comprehensive analyses of lottery tickets hypothesis on automatic speech recognition. The authors verified the existence of highly sparse “winning tickets” in ASR task, and analyze its robustness to noise, transferable to other datasets, and supports with structured sparsity. They conduct experiments on different model structures (CTC, RNN-T, Transfomer) and different datasets (LibriSpeech, TED-LIUM, Common Voice) with extensive analyses.  In this paper, authors propose to use lottery ticket hypothesis (LTH) for ASR model pruning. The whole idea generally inherits from the original LTH paper. The model is first trained from scratch. Then you collect all the model weights and sort them. A proportion of the smallest weights are set and fixed to 0. Given this sparsified subnetwork, one further repeats the whole process and prune more parameters. After several rounds, if the pruned subnetwork still gives at least the same performance of the original full network , it is called a "winning ticket". The "winning ticket" subnetwork should be lightweight, transferrable, and noise-robust. This work applies LTH to 3 models (CNN-RNN CTC, RNN-T, Conformer) on 3 datasets (TED-LIUM, Common Voice, LibriSpeech). The authors empirically show  1. The winning ticket exists for each method. The subnetwork could be only 4.4% size of the original full model, but still matches the performance. 2. Compared to other pruning methods, LTH performs best. 3. LTH with pretrained model performs better than LTH with random initialization.  Furthermore, the paper shows 1. One can enforce block sparsity for pruning to facilitate chip design. 2. The winning ticket can transfer to a new dataset and work well. 3. The pruned network is noise-robust.  This paper investigates using the lottery tickets hypothesis (LTH) strategy for pruning neural network weights for speech recognition. The method first explains the general LTH framework and extensions with transfer learning scenarios. The paper shows the effectiveness of the proposed method in the standard ASR tasks, pre-training with other models, and transfer learning, especially in noisy speech recognition tasks.  Some comments: - In the introduction "End-to-end automatic speech recognition (ASR) (Wang et al., 2019)": (Wang et al., 2019) is not a representative paper of end-to-end ASR - In the introduction, "For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020)": Again not sure (Kinoshita et al., 2020) is a correct citation. I think the following papers are more appropriate   - Haeb-Umbach, Reinhold, et al. "Speech processing for digital home assistants: Combining signal processing with deep-learning techniques." IEEE Signal processing magazine 36.6 (2019): 111-124.   - Kinoshita, Keisuke, et al. "The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech." 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. IEEE, 2013. - Currently, data augmentation techniques (artificially contaminate the clean speech with various noisy data) are more standard than transfer learning for noisy speech recognition. Please discuss it. - Section 2 "On the model level, compared to CV models, speech models are mostly based on RNN backbones": I'm confused about this description. As you mentioned in the previous paragraphs, many speech models now use transformer, conformer, and CNN. - Section 2 "with 10ms shift is 1,000": this is not exactly true as we usually use downsampling at the beginning of the encoder layer. - Section 3, RQ3: Frankly, I don't think this is an essential question for me. This sounds very trivial that the use of pre-trained models would improve the performance. - Section 4, What is the main message of Figure 1? I don't find the particular effectiveness of the proposed method from this figure/example. - Section 4, Table 3: Why are the other methods, especially KD-based methods, so worse? - Section 5 "A common way to address this issue is through speaker adaptation (Zhao, 1994;": Again, I don't think (Zhao, 1994) is a representative paper for speaker adaptation. MAP/MLLR adaptation techniques exist in this era, and they were standard techniques in the GMM era. - Section 5: I recommend the authors use the real noisy speech data like the CHiME data    The paper explores the lottery ticket hypothesis, i.e., the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models within the context of ASR. The authors show that it is possible to find subnetworks which contain just a fraction of the number of parameters of the full models without sacrificing performance on ASR. Three different architectures are considered and results reported on 3 datasets. Additional studies investigating the transferability and robustness of the sparse subnetworks are also conducted.
abstractText: Lightweight speech recognition models have seen explosive demands owing to a growing amount of speech-interactive features on mobile devices. Since designing such systems from scratch is non-trivial, practitioners typically choose to compress large (pre-trained) speech models. Recently, lottery ticket hypothesis reveals the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models. In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity. We conducted extensive experiments on CTC, RNN-Transducer, and Transformer models, and verified the existence of highly sparse “winning tickets” that can match the full model performance across those backbones. We obtained winning tickets that have less than 20% of full model weights on all backbones, while the most lightweight one only keeps 4.4% weights. Those winning tickets generalize to structured sparsity with no performance loss, and transfer exceptionally from large source datasets to various target datasets. Perhaps most surprisingly, when the training utterances have high background noises, the winning tickets even substantially outperform the full models, showing the extra bonus of noise robustness by inducing sparsity. All our codes will be released.
authors.name: ULTRA-LIGHTWEIGHT
authors.name: TRANSFERABLE
references.authors: Dario Amodei Sundaram Ananthanarayanan Rishita Anubhai Jingliang Bai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Qiang Cheng Guoliang Chen
references.title: Deep speech 2: End-to-end speech recognition in english and mandarin
references.venue: In ICML,
references.authors: Rosana Ardila Megan Branson Kelly Davis Michael Kohler Josh Meyer Michael Henretty Reuben Morais Lindsay Saunders Francis Tyers Gregor Weber
references.title: Common Voice: A massivelymultilingual speech corpus
references.venue: In Proceedings of The 12th Language Resources and Evaluation Conference,
references.authors: Sercan Ö Arık Markus Kliegl Rewon Child Joel Hestness Andrew Gibiansky Chris Fougner Ryan Prenger Adam Coates
references.title: Convolutional recurrent neural networks for small-footprint keyword spotting
references.venue: In INTERSPEECH,
references.authors: Alexei Baevski Henry Zhou Abdelrahman Mohamed Michael Auli
references.title: wav2vec 2.0: A framework for self-supervised learning of speech representations
references.venue: arXiv preprint arXiv:2006.11477,
references.authors: Dzmitry Bahdanau Jan Chorowski Dmitriy Serdyuk Philemon Brakel Yoshua Bengio
references.title: Endto-end attention-based large vocabulary speech recognition
references.venue: In ICASSP,
references.authors: Fadi Biadsy Ron J Weiss Pedro J Moreno Dimitri Kanvesky Ye Jia
references.title: Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation
references.venue: In INTERSPEECH,
references.authors: Davis Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag
references.title: What is the state of neural network pruning
references.venue: arXiv preprint arXiv:2003.03033,
references.authors: William Chan Navdeep Jaitly Quoc Le Oriol Vinyals
references.title: Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
references.venue: In ICASSP,
references.authors: Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Michael Carbin Zhangyang Wang
references.title: The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models
references.venue: arXiv preprint arXiv:2012.06908,
references.authors: Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin
references.title: The lottery ticket hypothesis for pre-trained BERT networks
references.venue: NeurIPS,
references.authors: Tianlong Chen Yu Cheng Zhe Gan Jingjing Liu Zhangyang Wang
references.title: Ultra-data-efficient GAN training: Drawing a lottery ticket first, then training it toughly
references.venue: arXiv preprint arXiv:2103.00397,
references.authors: Tianlong Chen Yongduo Sui Xuxi Chen Aston Zhang Zhangyang Wang
references.title: A unified lottery ticket hypothesis for graph neural networks
references.venue: arXiv preprint arXiv:2102.06790,
references.authors: Tianlong Chen Zhenyu Zhang Sijia Liu Shiyu Chang Zhangyang Wang
references.title: Long live the lottery: The existence of winning tickets in lifelong learning
references.venue: In ICLR,
references.authors: Xiaohan Chen Yu Cheng Shuohang Wang Zhe Gan Zhangyang Wang Jingjing Liu
references.title: Earlybert: Efficient bert training via early-bird lottery tickets
references.venue: arXiv preprint arXiv:2101.00063,
references.authors: Xuxi Chen Zhenyu Zhang Yongduo Sui Tianlong Chen
references.title: GANs can play lottery tickets too
references.venue: In ICLR,
references.authors: Chung-Cheng Chiu Tara N Sainath Yonghui Wu Rohit Prabhavalkar Patrick Nguyen Zhifeng Chen Anjuli Kannan Ron J Weiss Kanishka Rao Ekaterina Gonina
references.title: State-of-the-art speech recognition with sequence-to-sequence models
references.venue: In ICASSP,
references.authors: Jan Chorowski Dzmitry Bahdanau Dmitriy Serdyuk Kyunghyun Cho Yoshua Bengio
references.title: Attention-based models for speech recognition
references.venue: In ICONIP, pp
references.authors: Jordan Cohen
references.title: Embedded speech recognition applications in mobile phones: Status, trends, and challenges
references.venue: In ICASSP,
references.authors: Shrey Desai Hongyuan Zhan Ahmed Aly
references.title: Evaluating lottery tickets under distributional shifts
references.venue: arXiv preprint arXiv:1910.12708,
references.authors: Fenglin Ding Wu Guo Bin Gu Zhenhua Ling Jun Du
references.title: Adaptive speaker normalization for ctc-based speech recognition
references.venue: In INTERSPEECH,
references.authors: Linhao Dong Shuang Xu Bo Xu
references.title: Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Linhao Dong Shiyu Zhou Wei Chen Bo Xu
references.title: Extending recurrent neural aligner for streaming end-to-end speech recognition in Mandarin
references.venue: In INTERSPEECH,
references.authors: Erich Elsen Marat Dukhan Trevor Gale Karen Simonyan
references.title: Fast sparse convnets
references.venue: In CVPR,
references.authors: Utku Evci Fabian Pedregosa Aidan Gomez Erich Elsen
references.title: The difficulty of training sparse neural networks
references.venue: arXiv preprint arXiv:1906.10732,
references.authors: Florian Eyben Martin Wöllmer Björn Schuller Alex Graves
references.title: From speech to letters-using a novel neural network architecture for grapheme based ASR
references.venue: IEEE Workshop on Automatic Speech Recognition & Understanding,
references.authors: Jonathan Frankle Michael Carbin
references.title: The lottery ticket hypothesis: Finding sparse, trainable neural networks
references.venue: In ICLR, 2019a. URL https://openreview.net/forum?id=rJl-b3RcF7
references.authors: Jonathan Frankle Michael Carbin
references.title: The lottery ticket hypothesis: Finding sparse, trainable neural networks
references.venue: In ICLR,
references.authors: Jonathan Frankle Gintare Karolina Dziugaite Daniel Roy Michael Carbin
references.title: Linear mode connectivity and the lottery ticket hypothesis
references.venue: In ICML,
references.authors: Jonathan Frankle Gintare Karolina Dziugaite Daniel Roy Michael Carbin
references.title: Linear mode connectivity and the lottery ticket hypothesis
references.venue: In ICML,
references.authors: Jonathan Frankle David J. Schwab Ari S. Morcos
references.title: The early phase of neural network training
references.venue: In ICLR, 2020c. URL https://openreview.net/forum?id=Hkl1iRNFwS
references.authors: Trevor Gale Erich Elsen Sara Hooker
references.title: The state of sparsity in deep neural networks
references.venue: arXiv preprint arXiv:1902.09574,
references.authors: Dawei Gao Xiaoxi He Zimu Zhou Yongxin Tong Ke Xu Lothar Thiele
references.title: Rethinking pruning for accelerating deep inference at the edge
references.venue: In SIGKDD,
references.authors: J-L Gauvain Chin-Hui Lee
references.title: Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains
references.venue: IEEE transactions on speech and audio processing,
references.authors: Sharath Girish Shishira R Maiya Kamal Gupta Hao Chen Larry Davis Abhinav Shrivastava
references.title: The lottery ticket hypothesis for object recognition
references.venue: arXiv preprint arXiv:2012.04643,
references.authors: Alex Graves
references.title: Sequence transduction with recurrent neural networks
references.venue: arXiv preprint arXiv:1211.3711,
references.authors: Alex Graves Navdeep Jaitly
references.title: Towards end-to-end speech recognition with recurrent neural networks
references.venue: In ICML,
references.authors: Alex Graves Santiago Fernández Faustino Gomez Jürgen Schmidhuber
references.title: Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks
references.venue: In ICML, pp
references.authors: Alex Graves Abdel-rahman Mohamed Geoffrey Hinton
references.title: Speech recognition with deep recurrent neural networks
references.venue: In ICASSP,
references.authors: Scott Gray Alec Radford Diederik P Kingma
references.title: Gpu kernels for block-sparse weights
references.venue: arXiv preprint arXiv:1711.09224,
references.authors: Shupeng Gui Haotao Wang Chen Yu Haichuan Yang Zhangyang Wang Ji Liu
references.title: Model compression with adversarial robustness: A unified optimization framework
references.authors: Anmol Gulati James Qin Chung-Cheng Chiu Niki Parmar Yu Zhang Jiahui Yu Wei Han Shibo Wang Zhengdong Zhang Yonghui Wu
references.title: Conformer: Convolution-augmented transformer for speech recognition
references.authors: Reinhold Haeb-Umbach Shinji Watanabe Tomohiro Nakatani Michiel Bacchiani Bjorn Hoffmeister Michael L Seltzer Heiga Zen Mehrez Souden
references.title: Speech processing for digital home assistants: Combining signal processing with deep-learning techniques
references.venue: IEEE Signal processing magazine,
references.authors: Song Han Huizi Mao William J Dally
references.title: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
references.venue: arXiv preprint arXiv:1510.00149,
references.authors: Awni Hannun Carl Case Jared Casper Bryan Catanzaro Greg Diamos Erich Elsen Ryan Prenger Sanjeev Satheesh Shubho Sengupta Adam Coates
references.title: Deep speech: Scaling up end-to-end speech recognition
references.venue: arXiv preprint arXiv:1412.5567,
references.authors: Yanzhang He Tara N Sainath Rohit Prabhavalkar Ian McGraw Raziel Alvarez Ding Zhao David Rybach Anjuli Kannan Yonghui Wu Ruoming Pang
references.title: Streaming end-to-end speech recognition for mobile devices
references.venue: In ICASSP,
references.authors: Navdeep Jaitly Patrick Nguyen Andrew Senior Vincent Vanhoucke
references.title: Application of pretrained deep neural networks to large vocabulary speech recognition
references.venue: In INTERSPEECH,
references.authors: Ye Jia Ron J Weiss Fadi Biadsy Wolfgang Macherey Melvin Johnson Zhifeng Chen Yonghui Wu
references.title: Direct speech-to-speech translation with a sequence-to-sequence model
references.venue: In INTERSPEECH,
references.authors: Neha Mukund Kalibhat Yogesh Balaji Soheil Feizi
references.title: Winning lottery tickets in deep generative models
references.venue: arXiv preprint arXiv:2010.02350,
references.authors: Kwangyoun Kim Kyungmin Lee Dhananjaya Gowda Junmo Park Sungsoo Kim Sichen Jin Young-Yoon Lee Jinsu Yeo Daehyun Kim Seokyeong Jung
references.title: Attention based on-device streaming speech recognition with large speech corpus
references.venue: In ASRU,
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization, 2017
references.authors: Keisuke Kinoshita Marc Delcroix Takuya Yoshioka Tomohiro Nakatani Emanuel Habets Reinhold Haeb-Umbach Volker Leutnant Armin Sehr Walter Kellermann Roland Maas
references.title: The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech
references.venue: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics,
references.authors: Keisuke Kinoshita Tsubasa Ochiai Marc Delcroix Tomohiro Nakatani
references.title: Improving noise robust automatic speech recognition with single-channel time-domain enhancement network
references.venue: In ICASSP,
references.authors: Markus Kliegl Siddharth Goyal Kexin Zhao Kavya Srinet Mohammad Shoeybi
references.title: Trace norm regularization and faster inference for embedded speech recognition RNNs
references.venue: arXiv preprint arXiv:1710.09026,
references.authors: Christopher J Leggetter Philip C Woodland
references.title: Maximum likelihood linear regression for speaker adaptation of continuous density hidden markov models
references.venue: Computer speech & language,
references.authors: Chenxing Li Lei Zhu Shuang Xu Peng Gao Bo Xu
references.title: Compression of acoustic model via knowledge distillation and pruning
references.venue: In ICPR, pp. 2785–2790
references.authors: Jinyu Li Li Deng Yifan Gong Reinhold Haeb-Umbach
references.title: An overview of noise-robust automatic speech recognition
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Ke Li Jinyu Li Yong Zhao Kshitiz Kumar Yifan Gong
references.title: Speaker adaptation for end-to-end CTC models
references.venue: In SLT,
references.authors: Zerui Li Yue Ming Lei Yang Jing-Hao Xue
references.title: Mutual-learning sequence-level knowledge distillation for automatic speech
references.venue: recognition. Neurocomputing,
references.authors: Zhuohan Li Eric Wallace Sheng Shen Kevin Lin Kurt Keutzer Dan Klein Joey Gonzalez
references.title: Train big, then compress: Rethinking model size for efficient training and inference of transformers
references.venue: In ICML,
references.authors: Hank Liao
references.title: Speaker adaptation of context dependent deep neural networks
references.venue: In ICASSP,
references.authors: Hong Liu Mingsheng Long Jianmin Wang Michael I Jordan
references.title: Towards understanding the transferability of deep representations
references.venue: arXiv preprint arXiv:1909.12031,
references.authors: Zhuang Liu Mingjie Sun Tinghui Zhou Gao Huang Trevor Darrell
references.title: Rethinking the value of network pruning
references.venue: In ICLR,
references.authors: Haoyu Ma Tianlong Chen Ting-Kuei Hu Chenyu You Xiaohui Xie Zhangyang Wang
references.title: Good students play big lottery better
references.venue: arXiv preprint arXiv:2101.03255,
references.authors: Ian McGraw Rohit Prabhavalkar Raziel Alvarez Montse Gonzalez Arenas Kanishka Rao David Rybach Ouais Alsharif Haşim Sak Alexander Gruenstein Françoise Beaufays
references.title: Personalized speech recognition on mobile devices
references.venue: In ICASSP,
references.authors: Rahul Mehta
references.title: Sparse transfer learning via winning lottery tickets
references.venue: arXiv preprint arXiv:1905.07785,
references.authors: Zhong Meng Yashesh Gaur Jinyu Li Yifan Gong
references.title: Speaker adaptation for attention-based end-to-end speech recognition
references.venue: In INTERSPEECH,
references.authors: Yajie Miao Mohammad Gowayyed Florian Metze
references.title: EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
references.venue: In ASRU,
references.authors: Ari Morcos Haonan Yu Michela Paganini Yuandong Tian
references.title: One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers
references.venue: In NeurIPS,
references.authors: Sharan Narang Erich Elsen Gregory Diamos Shubho Sengupta
references.title: Exploring sparsity in recurrent neural networks
references.venue: arXiv preprint arXiv:1704.05119,
references.authors: Sharan Narang Eric Undersander Gregory Diamos
references.title: Block-sparse recurrent neural networks
references.venue: arXiv preprint arXiv:1711.02782,
references.authors: Vassil Panayotov Guoguo Chen Daniel Povey Sanjeev Khudanpur
references.title: Librispeech: An ASR corpus based on public domain audio books
references.venue: In ICASSP,
references.authors: Jinhwan Park Yoonho Boo Iksoo Choi Sungho Shin Wonyong Sung
references.title: Fully neural network based speech recognition on mobile and embedded devices
references.venue: In NeurIPS,
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga
references.title: Pytorch: An imperative style, high-performance deep learning
references.venue: library. NeurIPS,
references.authors: Joelle Pineau Philippe Vincent-Lamarre Koustuv Sinha Vincent Lariviere Alina Beygelzimer Florence d’Alché Buc Emily Fox Hugo Larochelle
references.title: Improving reproducibility in machine learning research
references.venue: Journal of Machine Learning Research,
references.authors: Rohit Prabhavalkar Tara N Sainath Yonghui Wu Patrick Nguyen Zhifeng Chen Chung-Cheng Chiu Anjuli Kannan
references.title: Minimum word error rate training for attention-based sequence-to-sequence models
references.venue: In ICASSP,
references.authors: Sai Prasanna Anna Rogers Anna Rumshisky
references.title: When BERT plays the lottery, all tickets are winning
references.venue: arXiv preprint arXiv:2005.00561,
references.authors: Kanishka Rao Haşim Sak Rohit Prabhavalkar
references.title: Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-Transducer
references.venue: In ASRU,
references.authors: Alex Renda Jonathan Frankle Michael Carbin
references.title: Comparing rewinding and fine-tuning in neural network pruning
references.venue: In ICLR,
references.authors: Anthony Rousseau Paul Deléglise Yannick Esteve
references.title: TED-LIUM: an automatic speech recognition dedicated corpus
references.venue: In LREC, pp
references.authors: Tara N Sainath Yanzhang He Bo Li Arun Narayanan Ruoming Pang Antoine Bruguier Shuo-yiin Chang Wei Li Raziel Alvarez Zhifeng Chen
references.title: A streaming on-device end-to-end model surpassing server-side conventional model quality and latency
references.venue: In ICASSP,
references.authors: Pedro Savarese Hugo Silva Michael Maire
references.title: Winning the lottery with continuous sparsification
references.venue: In NeurIPS,
references.authors: Johan Schalkwyk Doug Beeferman Françoise Beaufays Bill Byrne Ciprian Chelba Mike Cohen Maryam Kamvar Brian Strope
references.title: Your word is my command”: Google search by voice: A case study
references.venue: In Advances in speech recognition,
references.authors: Romain Serizel Nicolas Turpault Ankit Shah Justin Salamon
references.title: Sound event detection in synthetic domestic environments
references.venue: In ICASSP,
references.authors: Yuan Shangguan Jian Li Qiao Liang Raziel Alvarez Ian McGraw
references.title: Optimizing speech recognition for the edge
references.venue: arXiv preprint arXiv:1909.12408,
references.authors: Jonathan Shen Ruoming Pang Ron J Weiss Mike Schuster Navdeep Jaitly Zongheng Yang Zhifeng Chen Yu Zhang Yuxuan Wang Rj Skerrv-Ryan
references.title: Natural TTS synthesis by conditioning Wavenet on Mel spectrogram predictions
references.venue: In ICASSP,
references.authors: Khe Chai Sim Petr Zadrazil Françoise Beaufays
references.title: An investigation into on-device personalization of end-to-end automatic speech recognition models
references.venue: In INTERSPEECH,
references.authors: Ryoichi Takashima Sheng Li Hisashi Kawai
references.title: An investigation of a knowledge distillation method for ctc acoustic models
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Ryu Takeda Kazuhiro Nakadai Kazunori Komatani
references.title: Node pruning based on entropy of weights and node activity for small-footprint acoustic model based on deep neural networks
references.venue: In INTERSPEECH,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in neural information processing systems,
references.authors: Alex Waibel Ahmed Badran Alan W Black Robert Frederking Donna Gates Alon Lavie Lori Levin Kevin A Lenzo Laura Mayfield Tomokiyo Jurgen Reichert
references.title: Speechalator: Two-way speech-to-speech translation on a consumer PDA
references.venue: In Eighth European Conference on Speech Communication and Technology,
references.authors: Chaoqi Wang Guodong Zhang Roger Grosse
references.title: Picking winning tickets before training by preserving gradient flow
references.venue: In ICLR,
references.authors: Dong Wang Xiaodong Wang Shaohe Lv
references.title: An overview of end-to-end automatic speech recognition. Symmetry
references.authors: Yongqiang Wang Abdelrahman Mohamed Due Le Chunxi Liu Alex Xiao Jay Mahadeokar Hongzhao Huang Andros Tjandra Xiaohui Zhang Frank Zhang
references.title: Transformer-based acoustic modeling for hybrid speech recognition
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Yuxuan Wang RJ Skerry-Ryan Daisy Stanton Yonghui Wu Ron J Weiss Navdeep Jaitly Zongheng Yang Ying Xiao Zhifeng Chen Samy Bengio
references.title: Tacotron: Towards end-to-end speech synthesis
references.venue: In INTERSPEECH,
references.authors: Wei Wen Yuxiong He Samyam Rajbhandari Minjia Zhang Wenhan Wang Fang Liu Bin Hu Yiran Chen Hai Li
references.title: Learning intrinsic sparse structures within long short-term memory
references.venue: In ICLR,
references.authors: Felix Weninger Jesús Andrés-Ferrer Xinwei Li Puming Zhan
references.title: Listen, attend, spell and adapt: Speaker adapted sequence-to-sequence ASR
references.venue: In INTERSPEECH,
references.authors: Phil C Woodland
references.title: Speaker adaptation for continuous density hmms: A review
references.venue: In ISCA Tutorial and Research Workshop (ITRW) on Adaptation Methods for Speech Recognition,
references.authors: Zhaofeng Wu Ding Zhao Qiao Liang Jiahui Yu Anmol Gulati Ruoming Pang
references.title: Dynamic sparsity neural networks for automatic speech recognition
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Shaokai Ye Kaidi Xu Sijia Liu Hao Cheng Jan-Henrik Lambrechts Huan Zhang Aojun Zhou Kaisheng Ma Yanzhi Wang Xue Lin
references.title: Adversarial robustness vs. model compression, or both
references.authors: Ji Won Yoon Hyeonseung Lee Hyung Yong Kim Won Ik Cho Nam Soo Kim
references.title: Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Haoran You Chaojian Li Pengfei Xu Yonggan Fu Yue Wang Xiaohan Chen Richard G. Baraniuk Zhangyang Wang Yingyan Lin
references.title: Drawing early-bird tickets: Toward more efficient training of deep networks
references.venue: In ICLR,
references.authors: Haonan Yu Sergey Edunov Yuandong Tian Ari S Morcos
references.title: Playing the lottery with rewards and multiple languages: Lottery tickets in RL and NLP
references.venue: In ICLR,
references.authors: Heiga Zen Viet Dang Rob Clark Yu Zhang Ron J Weiss Ye Jia Zhifeng Chen Yonghui Wu
references.title: LibriTTS: A corpus derived from LibriSpeech for Text-to-Speech
references.venue: In INTERSPEECH,
references.authors: Shunshi Zhang Bradly C Stadie
references.title: One-shot pruning of recurrent neural networks by jacobian spectrum evaluation
references.venue: In ICLR,
references.authors: Yu Zhang William Chan Navdeep Jaitly
references.title: Very deep convolutional networks for end-to-end speech recognition
references.venue: In ICASSP,
references.authors: Guanlong Zhao Shaojin Ding Ricardo Gutierrez-Osuna
references.title: Foreign accent conversion by synthesizing speech from phonetic posteriorgrams
references.venue: In INTERSPEECH,
sections.heading: 1 INTRODUCTION
sections.text: End-to-end automatic speech recognition (ASR) (Wang et al., 2019; Hannun et al., 2014; Graves, 2012; Chorowski et al., 2015; Dong et al., 2018a) has become an indispensable technology in consumer-interactive devices (e.g., smartphones, smart speakers, tablets) over the past few years (He et al., 2019; Cohen, 2008; Schalkwyk et al., 2010). Conventional on-device ASR system usually requires the involvement of servers, i.e., streaming the audio to servers and then streaming the results back to the devices. By contrast, recent studies (He et al., 2019; McGraw et al., 2016; Kim et al., 2019; Sim et al., 2019; Park et al., 2018; Waibel et al., 2003; Arık et al., 2017) have spurred the success of ASR models fully run on devices, which can be advantageous in terms of computational resources, latency, and user data privacy.
  Developing on-device ASR models are challenging since the computational resources (e.g., CPU, memory, battery) are typically very limited. A standard design approach to fit ASR model in budget is through applying various neural network compression techniques to the larger ASR models, such as network pruning (Takeda et al., 2017; Shangguan et al., 2019; Gao et al., 2020), knowledge distillation (Li et al., 2018a), and parameter quantization (He et al., 2019; Sainath et al., 2020). However, there always exists a trade-off between computational efficiency and the model performance, and the efficiency improvements are usually at the cost of word error rate (WER). In most prior works, we observed a non-negligible degradation of WER in compressed models.
  A new horizon emerges with the discovery of lottery tickets hypothesis (LTH) (Frankle & Carbin, 2019b). LTH empirically demonstrated the existence of highly sparse matching subnetworks (i.e., winning tickets) in full dense networks, that can be independently trained from scratch to match or even surpass the performance of the latter. LTH was widely observed in various models in computer vision (Liu et al., 2019b; Evci et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al.,
  2020a; You et al., 2020; Ma et al., 2021; Girish et al., 2020) and natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna et al., 2020; Chen et al., 2020b;c). Yet to our best knowledge, it has not been studied nor utilized in the realm of speech processing and recognition .
  This paper presents the first investigation on LTH for developing on-device ASR models. Despite the rich literature of LTH in vision and language, a practically useful winning ticket for real-world ASR would demand two unique properties: transferability, noise robustness. • As one persistent challenges of ASR, each individual has a different voice and speaking style.
  Unlike text or images, whose data are much more “standardized”, the spoken word varies greatly based on regional dialects, speed, emphasis, even social class and gender. Therefore, scaling up any ASR system has always been a significant obstacle, since the testing utterances may have very different distribution from the training utterances. That has made transferability a crucial demand for ASR in the open world.
  • In an ideal world, one would have to speak very clearly, slowly, and in an environment with no background noise, for the sounds being recognized: that unfortunately will not happen in the practice. In the real-world ASR applications, noise robustness is becoming another crucially demanded technological factor since ASR is now expected to work in much more difficult acoustic environments than in the past (Li et al., 2014). For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020; Haeb-Umbach et al., 2019; Kinoshita et al., 2013). Even in the standard ASR benchmark such as LibriSpeech (Panayotov et al., 2015), there are significant background noise in its “clean” subset (Zen et al., 2019).
  More importantly, prior LTH studies mostly use unstructured sparsity during model pruning. However, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU Kernels (Gray et al., 2017)). Together with the two unique gaps, they make ASR no less challenging than visual recognition or text understanding, if not more. They account for the prior arts’ difficulty to maintain a satisfactory balance between model efficiency and recognition performance; and similarly, they question the applicability of LTH in ASR.
  We provide an affirmative, positive answer of LTH in ASR. As the subject of study, we choose most commonly used ASR architectures in both research and products over the past few years: 1) CNN+RNN with connectionist temporal classification (CTC) (Graves et al., 2006); 2) RNNTransducer (Graves, 2012); and 3) Convolution-augmented Transformer (Gulati et al., 2020). We conducted extensive LTH experiments using these backbones on three popular corpora: TEDLIUM (Rousseau et al., 2012), Common Voice (Ardila et al., 2020), and LibriSpeech (Panayotov et al., 2015). Unlike most of existing LTH studies that only approached to explaining and demonstrating the correctness of LTH theory, in this work, we make the first attempt to apply LTH to real-world use cases. Namely, we investigate three unique properties that were rarely studied in previous LTH research but are key to user-interactive ASR devices, bringing new insights to both LTH theory and lightweight ASR research. Our main contributions are outlined below:
  • We for the first time reveal the existence of winning tickets in the context of ASR by answering three research questions regarding LTH theory. The most lightweight winning tickets from CTC, RNN-Transducer, and Conformer backbones only possess 21.0%, 10.7%, and 8.6% remaining nonzero weights, respectively. We also show that winning tickets significantly outperform other state-of-the-art network pruning and knowledge distillation methods.
  • We are the first to explore the use of structured sparsity (i.e., block sparsity (Narang et al., 2017b)) in LTH, and successfully found highly sparse winning tickets (e.g., 4.4% remaining weights) that have no performance degradation compared to using unstructured sparsity.
  • Winning tickets have exceptional transferability across different datasets, which are notably better than full models. Also, the winning tickets identified from large source datasets transfer better.
  • In the presence of various levels of background noise, the winning tickets consistently achieve significantly better WERs than full models. That indicates stronger noise robustness might be an extra bonus of inducing sparsity, and a missing gem by previous LTH works.
sections.heading: 2 RELATED WORKS
sections.text: End-to-End Automatic Speech Recognition. Previous mainstream ASR systems are mostly based on hidden Markov model (HMM)-Gaussian mixture model (GMM) or HMM-Deep neural network
  (DNN). These systems can achieve descent performance. However, such system is usually composed of modules (e.g., acoustic model, language model, lexicon) that are needed to be trained separately, which makes it hard to be optimize globally in products (Wang et al., 2019).
  End-to-end ASR can directly transcribe an input audio sequence to a token (e.g., grapheme or phoneme) sequence. Current end-to-end ASR frameworks can be generally categorized into three types: CTC-based (Hannun et al., 2014; Amodei et al., 2016; Graves & Jaitly, 2014; Miao et al., 2015; Eyben et al., 2009), RNN-transducer (Graves, 2012; Graves et al., 2013; Rao et al., 2017; Dong et al., 2018b), sequence-to-sequence (Seq2Seq) model (Chorowski et al., 2015; Bahdanau et al., 2016; Chan et al., 2016; Zhang et al., 2017; Chiu et al., 2018; Prabhavalkar et al., 2018), and Transformer model (Gulati et al., 2020; Dong et al., 2018a; Wang et al., 2020b; Baevski et al., 2020).
  CTC (Graves et al., 2006) is essentially a loss function, which maximizes the probability of all the paths that correspond to the ground-truth token sequence, with an augmented blank token “-” indicating no output. It avoids the need of segmental alignment/labels in training utterances, which makes tons of speech materials usable without additional annotating effort and thus fully exploits the modeling capacity of DNNs. RNN-transducer is composed of an encoder for the input audio sequence, a prediction network to model the interdependencies in between the output token sequence, and a joint network to align the input and output sequence and produce the prediction. Seq2Seq model usually has an encoder-decoder structure, where the encoder maps the input audio sequence to a hidden representation sequence and the decoder autogressively decodes the output token sequence. An attention mechanism (Chorowski et al., 2015) is trained to learn the alignment between the input and output sequences. Transformer models are similar to Seq2seq models but use multi-head attention (MHA) (Vaswani et al., 2017) layers for encoders and decoders, which has been proven to achieve the state-of-the-art ASR performance. Lottery Tickets Hypothesis. The recently emerged LTH (Frankle & Carbin, 2019b) deviates from the common wisdom of after-training pruning (Han et al., 2015), and demonstrates the existence of highly sparse subnetworks that are independently trainable from scratch, called winning tickets. Once trained, they are capable of matching or even surpassing the performance of their full models. Follow-up efforts (Renda et al., 2020; Frankle et al., 2020b) introduce the effective weight rewinding techniques to scale up LTH to large networks on large-scale datasets. LTH draws wide attention from various deep learning fields, and has been studied in image classification (Liu et al., 2019b; Evci et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al., 2020a; You et al., 2020; Ma et al., 2021; Chen et al., 2020a), natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna et al., 2020; Chen et al., 2020b;c), object detection (Girish et al., 2020), generative adversarial networks (Chen et al., 2021d; Kalibhat et al., 2020; Chen et al., 2021a), graph neural networks (Chen et al., 2021b), reinforcement learning (Yu et al., 2020), and life-long learning (Chen et al., 2021c). Several pioneer works (Mehta, 2019; Morcos et al., 2019; Desai et al., 2019; Chen et al., 2020b;a) also investigate LTH transferability across datasets and downstream tasks.
  Yet to the best of our knowledge, LTH in speech models remains untouched – and that would not be a trivial extension for three aspects of reasons. On the task level, ASR has unique demands for individual user transferrability and noise robustness, which has been explained previously. On the model level, compared to CV models, speech models are mostly based on RNN backbones (Hannun et al., 2014; Chan et al., 2016), which contain recursive computational graphs and are notoriously unstable to train. That makes the pruning of RNN-based models challenging too (Zhang & Stadie, 2019), and off-the-shelf methods developed for pruning CNNs are often found ineffective or even inferior to random pruning, when applied to RNNs. Pruning methods customized for RNNs typically call for special sparse structures or stability regularizations (Narang et al., 2017a;b; Kliegl et al., 2017; Wen et al., 2018; Zhang & Stadie, 2019), and it is hence unclear whether IMP would remain stable and effective for RNN-based models in ASR. On the data level, compared to NLP models, the sequence lengths of speech signals are usually significantly larger than word embeddings (e.g., the spectrogram sequence length of a 10-second speech utterance extracted with 10ms shift is 1,000; the number of words in a sentence is usually between 10 and 50), which also inevitably costs higher computational complexity. Therefore, finding sparse subnetworks that can maintain the full model performance for ASR models is practically meaningful yet highly non-trivial.
sections.heading: 3 PRELIMINARIES AND SETUPS
sections.text: Backbone Network. We investigate three ASR backbone networks that are widely used in both academia and productions: 1) CNN+LSTM (Amodei et al., 2016) model with CTC loss (Graves
  et al., 2006); 2) RNN-Transducer (Graves, 2012); and 3) Convolution-augmented Transformer (Conformer) (Gulati et al., 2020). Please see Appendix A.1 for details of the backbones.
  Datasets, Training, and Evaluations. We conducted experiments on three commonly used ASR corpora: TED-LIUM (Rousseau et al., 2012) (118 hours), Common Voice (Ardila et al., 2020) (582 hours), and LibriSpeech (Panayotov et al., 2015) (960 hours). Note that LibriSpeech has two test sets: test-clean – little noise inutterances; test-other – considerable noise in utterances. We test the LibriSpeech model on the two test sets hereinafter. During training, we set the batch size to 32 and an initial learning rate to 0.0003, which is annealed down by a factor of 1.1 at the end of each epoch. All the models were trained for 16 epochs. To evaluate the performance, we consider two measurements:
  • Word Error Rate. WER is the standard metric measuring the accuracy of ASR models. WER is computed as: WER = (S + I +D)/N , where S, I , D, and N denote the number of substitutions, insertions, deletions, and the total number of words, respectively.
  • Number of Parameters. The number of parameters measures the complexity of a model. In our case, since all subnetworks were pruned from the full models, we use the percentage of Remaining Weights as an alternative measurement. We define Sparsity as Sparsity(%) = 100%−Remaining Weights(%). Subnetworks. For a dense model f(x; θ), its subnetworks can be derived as f(x;m θ) with a binary pruning mask m ∈ {0, 1}d, where is the element-wise product and d is the dimension of pruneable model parameters. We use ADt (f(x; θ)) to represent the training algorithm (e.g., Adam (Kingma & Ba, 2017) with grid searched hyperparameters) that trains a network f(x; θ) on a dataset D (e.g., LibriSpeech) for t iterations. Let θ0 be the random initialized network weights. Subnetwork Evaluation. To measure the generalization ability of obtained subnetworks, we define ED(ADt (f(x; θ))) as the evaluation function of model f returned from ADt on the dataset D. Then, we further introduce:
  ? Matching subnetworks. Following the definition in (Frankle et al., 2020a; Chen et al., 2020b;a), a subnetwork f(x;m θ) is matching if it satisfies the following condition that indicates matching subnetworks achieve no worse performance than its dense counterpart under the same training algorithm ADt and evaluation metric ED: ED ( ADt (f (x;m θ)) ) ≥ ED ( ADt (f (x; θ0)) ) .
  ? Winning ticket. f(x;m θ) is a winning ticket for ADt , if it is (i) a matching subnetwork and (ii) θ = θ0 for ADt . ? Transferable Winning ticket. A subnetwork f(x;m θ) is transferable to target datasets {Di}Ni=1 if and only if it is a winning ticket for each ADiti . The subnetwork f(x;m θ) is derived from the source dataset Ds /∈ {Di}Ni=1. Pruning Method for Subnetwork Searching. Iterative weight magnitude pruning (IMP) is the widely used algorithm in previous LTH literature (Frankle & Carbin, 2019a; Frankle et al., 2020a; Chen et al., 2020b). To identify subnetworks f(x;m θ), IMP performs following three steps: (1) training a unpruned dense network to completion on a datasetD (i.e., applyingADt ); (2) eliminating a portion of insignificant weights with the globally smallest magnitudes (Han et al., 2015; Renda et al., 2020) so that the model only has si% of weights remaining (i.e., the sparsity); (3) rewinding model weights to θ (θ = θ0, the original random initialization; or θ = θpre, the weights from a pre-trained model) and finetuning the subnetwork to converge by leveraging ADt . Note that steps (2) and (3) usually needs to be iteratively repeated for several rounds for finding highly competitive winning tickets. In all experiments, we set si% = (1− 0.8i)× 100%, where i is the number of iterations.
sections.heading: 4 THE EXISTENCE OF WINNING TICKETS IN SPEECH RECOGNITION
sections.text: In this section, we explore the existence of winning tickets in the three ASR backbones. Namely, we would like to answer the following research questions from an empirical perspective:
  • RQ1: Can we find winning tickets f(x;mIMP θ) for speech recognition model using IMP? How much do they improve model complexity?
  • RQ2: Do winning tickets identified by IMP have less complexities or better performance, compared to random pruning/random tickets and other compression methods?
  • RQ3: Instead of using randomized weights θ0 as the initialization of IMP, does it improve the performance of winning tickets if we use weights θpre from a pre-trained model?
  RQ1: Does winning tickets exist in speech recognition models? To answer the questions, we conducted experiments on three backbones and three datasets. For each trial, we first run IMP to extract a binary pruning mask at each sparsity. Then, we generate one subnetwork at each sparsity by applying the corresponding mask to the model and reset the weights to the original random initialization θ0. Finally, we train each subnetwork and computes their WER on the test set to determine if they are winning tickets. All the training hyperparameters in training a subnetwork are the same as those in training the full model.
  As shown in Table 1, winning tickets can be identified on all three backbones. The most lightweight winning tickets on CTC, RNN-Transducer, and Conformer have 21.0%, 10.7%, and 8.6% remaining weights, respectively. In addition, we noticed that the RNN-Transducer and Conformer subnetworks at the extreme sparsity have less percentage of remaining weights than CTC subnetworks, likely due to these models being more over-parameterized (CTC: 86.62M parameters; RNN-Transducer: 132.23M; Conformer: 118.40M). These results show that, for a fixed dataset, the winning tickets extracted from larger models are sparser.
  Similarly, winning tickets can also be identified on all three datasets, as shown in Table 2. We also found that the sparsity of a winning ticket is correlated to the size of the dataset. For example, TEDLIUM has a relatively small size (118 hours) compared to CommonVoice (582 hours) and LibriSpeech (960 hours). Accordingly, the remaining weights of TED-LIUM winning ticket are significantly lower than that of CommonVoice and LibriSpeech winning tickets. A possible explanation is models become relatively more overparameterized for smaller training sets, which allows them to be more amenable for sparsification (Li et al., 2020). Similar observations can be found in (Chen et al., 2020b).
  From Table 1 and 2, another interesting finding is that the subnetworks with low sparsity (remain most of the weights) always achieve preferable performance than the full model. We also provided a visualization of the outputs from the full model, the most sparse subnetwork, and the best performing subnetwork in Figure 1 (see Appendix A.4 for more examples.). Similarly, we observed larger performance improvement on smaller datasets, possibly also due to the over parameterization issue. Results indicate that LTH is also a potential research direction in improving overall ASR performance.
  RQ2: Does IMP winning tickets have lower complexity or better performance than random pruning/tickets and other compression methods? As suggested in previous studies (Frankle & Carbin, 2019a; Chen et al., 2020b; 2021d; 2020a), the two key aspects for a winning ticket to achieve the desired performance are: 1) initial weight θ, and 2) mask generated from IMP mIMP . In this subsection, we test if this argument hold in ASR winning tickets. To achieve this, we compare the winning tickets against two baseline pruning approaches: random pruning and random tickets. The subnetworks identified with random pruning are initialized with θ0 but the masks are randomly generated mRAND. By contrast, the subnetworks identified with random tickets are randomly initialized θ ′
  0 but the masks are generated by IMP mIMP . As a result, with these two comparisons, we can explore if the two aspects are necessary to a winning ticket.
  We tested the three pruning approaches on TED-LIUM dataset. Figure 2 shows the WER curves of the three approaches. From the figure, we found that random pruning and random ticket could find
  winning tickets, but the extreme remaining weights are much higher than the one identified by IMP (IMP: 4.4%, Random Pruning: 26.2%, Random Ticket: 10.7%). Random pruning can roughly retain its performance when the remaining weights are larger than 26.2%. However, the performance drops dramatically as the remaining weights continue decreasing. Random ticket achieves very similar performance to IMP when the remaining weights are larger than 10.7%, but it degrades much faster than IMP after this point. Results verified the previous statement in ASR models that both initial weight θ and mask generated from IMP mIMP are necessary for winning tickets. Additionally, the comparison between random pruning and random ticket suggests that mIMP is more important than θ for winning tickets.
  Additionally, we compared our proposed approach against four state-of-the-art neural network compression baselines: 1) Standard Pruning (Han et al., 2015; Blalock et al., 2020; Shangguan et al., 2019); 2) TutorNet (Yoon et al., 2021); 3) MLKD + multi-teacher (Li et al., 2021); 4) Sequence-level KD (Takashima et al., 2018). Baseline 1 is the most commonly used and best performing network pruning approach, which iteratively prunes the lowest magnitude weights and train the network until reaching the target sparsity (without any rewinding). Baseline 2, 3, and 4 are
  three recent knowledge distillation approaches. As shown in Table 3, our winning ticket at extreme sparsity (Proposedext) achieves superior WER than all the baselines, while using less parameters.
  RQ3: Can IMP find better subnetworks by initializing from a pre-trained model? Training ASR model from a good initialization usually results in more satisfactory performance (Jaitly et al., 2012), which is commonly achieved by finetuning a pre-trained model in practice. Being aware of this, we would like to investigate if IMP can find better subnetworks for ASR by initializing with weights from a pre-trained model θ = θpre. To verify this, we ran IMP on CNN+LSTM backbone with TED-LIUM dataset, where the weights were initialized from either LibriSpeech pre-trained model θpre = θLibri or CommonVoice pre-trained model θpre = θCV .
  WER curves of the subnetworks in this experiment are shown in Figure 3. Both initializing from θLibri and θCV significantly improves the performance of the subnetworks at any sparsity. More importantly, we found a rapid WER increase when the remaining weights are less than 4.4% for IMP initialized from θ0. In contrast, the WERs of subnetworks in θLibri and θCV systems degrades much slower when the remaining weights are extremely low, indicating that the subnetworks that are identified from pre-trained models can utilize the parameters more efficiently. A possible reason is that the flatter loss surfaces of the pre-trained models (Liu et al., 2019a) make it more traceable for compression, leading to highly quality subnetworks, as also evidenced in (Chen et al., 2020b;a).
  Table 4 shows quantitative results of these systems. Initializing from θLibri achieves the best performance among all three systems. The WERs of subnetworks initialized from either θLibri or θCV at different sparsities varies marginally, which makes it hard to find the winning ticket at extreme sparsity. Therefore, we show the WER of all the system with 4.4% remaining weights instead, corresponding to the extreme sparsity of the subnetworks initialized with θ0. Although WER4.4% of θLibri and θCV
  systems are equal or higher than WERs of the corresponding full models, they are still significantly lower than θ0 system. Additionally, the remaining weights of best performing subnetworks in θLibri and θCV systems are higher than θ0, however, when reducing the amount of remaining weights, the WERs only have minimal degradations.
  Summary. We conducted extensive experiments to answer the three research questions about the existence of winning tickets in three ASR backbones. First, our results verified the existence of winning tickets in CTC, RNN-Transducer, and Conformer models, even at high sparsity (e.g., 4.4% remaining weights). Second, we compared IMP with random pruning/tickets and other state-of-the-art network compression methods. The results suggest that matching subnetworks extracted by IMP significantly exceed those extracted by random pruning and random tickets in all measurements, which corroborates that both binary pruning mask and weight initialization are indispensable in finding the winning tickets. Our approach also significantly outperforms other compression approaches, achieving state-of-the-art performance on ASR model compression. Lastly, we explored the use of weights from pre-trained models to initialize IMP. We found that, in this way, IMP can identify more effective winning tickets compared to random initialization, and more importantly, these winning tickets have a considerably higher parameter efficiency. These results collectively advocate the profound benefits that LTH can bring to both server-side and on-device ASR.
sections.heading: 5 TOWARDS PRACTICAL ASR WITH WINNING TICKETS: STUDYING STRUCTURED SPARSITY, TRANSFERABLITY, AND NOISE ROBUSTNESS
sections.text: After we proved the existence of winning tickets in ASR models, three more properties remain to be verified: structured sparsity, transferability, and noise robustness, which are key to ASR applications.
  Study of structured sparsity As we mentioned earlier, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU Kernels (Gray et al., 2017)). Therefore, it is critical to verify if we can also identify winning tickets using structured pruning. However, the exploration of structured pruning in prior LTH studies is very limited. (You et al., 2020) is only able to identify limited winning tickets with structured sparsity at low sparsity levels (70% remaining weights), which again emphasizes the difficulty in finding computation-friendly sparse patterns.
  In this experiment, we applied block sparsity (Narang et al., 2017b; Shangguan et al., 2019; Wu et al., 2021) with 1×4 block to subnetwork searching, and then evaluated the subnetworks to see if they can match the full model
  performance. Results in Table 5 show that using block sparsity during subnetwork searching process lead to comparable results to using unstructured pruning. Also, we visualized the weights of winning ticket models that were discovered with unstructured sparsity and block sparsity, as shown in Figure 4. We can clearly observe 1×4 block patterns from the weight matrix pruned with block sparsity. Study of Transferability In practical scenarios, the testing utterances are directly recorded from users in the wild, which may have very different distributions from the training utterances. A common way to address this issue is through speaker adaptation (Gauvain & Lee, 1994; Leggetter & Woodland, 1995; Woodland, 2001; Liao, 2013; Li et al., 2018b; Meng et al., 2019; Weninger et al., 2019), which is usually achieved by finetuning a pre-trained model on utterances whose distributions are similar to test utterances directly, or with additional loss terms (Li et al., 2018b; Meng et al., 2019; Ding et al., 2020). In the context of winning tickets, these sparse architectures have to be transferable to new datasets so that we are able to perform speaker adaptation on these models.
  To examine the transferability, we conducted the following experiment: First, we identify subnetworks f(x;mIMP θ0) on a source dataset Ds at different sparsities using IMP. Then, we re-train each subnetwork on a target dataset Dt and evaluate their performance. We tested the transferability between TED-LIUM, CommonVoice, and LibriSpeech datasets. These datasets are different in terms of recording scenario, noise level, speaker coverage, training set size, etc., and therefore, their utterances have very difference distributions. For example, TED-LIUM is created using TED talks, CommonVoice is composed of volunteer’s voice recorded through website or mobile apps, and LibriSpeech is extracted from LibriVox audio books.
  Figure 5 shows the performance of subnetworks transferring to the three target datasets. Results on the three datasets consistently suggest that the winning tickets are transferable across different datasets. When the remaining weights are larger than around 26.2%, the transferring tickets can generally replicate the performance of the winning tickets identified on the target datasets. However, at extremely high sparsity, the performance of transferring tickets degrades faster than the winning tickets identified on the target datasets, corresponding to the observations from previous studies (Chen et al., 2020c; Morcos et al., 2019; Chen et al., 2021d). For example, Table 9 in Appendix A.6 shows WER and remaining weight of CommonVoice and LibriSpeech winning tickets transferring to TED-LIUM dataset at extreme sparsity. Although the remaining weights of transferring winning tickets are higher than TED-LIUM winning ticket, they are still significantly lower than the full model. We also noticed is that the winning tickets identified from a larger dataset usually have better transferability, which is in accordance with (Morcos et al., 2019).
  Study of Noise Robustness The training/adaptation speech utterances are mostly collected from users, which are usually recorded from uncontrolled environments with notable background noise. Even in standard ASR benchmarks such as LibriSpeech (Panayotov et al., 2015), there are significant background noise even in its “clean” subset (Zen et al., 2019). To test the noise robustness of winning tickets, we conducted an experiment on TED-LIUM dataset. Namely, we re-trained the winning tickets identified from TED-LIUM, CommonVoice, and LibriSpeech on resynthesized TED-
  LIUM dataset with background noise. We used DESED dataset (Serizel et al., 2020) as the noise source, which consists of various background sounds in domestic environments such as bell, animals, running water, speech, vacuum cleaners, etc. During resynthesis, the noise audio is added to the original speech utterance with a random noise level, which is drawn from a uniform distribution from [0, Nmax]. In addition, we considered three different noise levels: Nmax = 0 (no noise),Nmax = 0.2 (low noise), Nmax = 0.5 (high noise).
  Table 6 shows WERs on TED-LIUM test set of the re-trained winning tickets. First, we found the performance of the full model is dramatically susceptible to background noise. When the noise level is increase from Nmax = 0 to Nmax = 0.5, WER increases from 15.93% to 38.21%. In contrast, WERs of TED-LIUM winning ticket at different noise level are very close, indicating that the sparse subnetwork is substaintially more robust to noise. Although the winning tickets from CommonVoice and LibriSpeech cannot reach the WER of TED-LIUM winning tickets on noise data, their WERs are
  still significantly lower than full model. More essentially, we can adapt transfer tickets to any target users without the need of re-finding winning tickets, which is more feasible for practical scenarios. Our findings coincide with the ones in (Ye et al., 2019; Gui et al., 2019) that an appropriate sparsity connectivity severs as implicit regularization for network training, which produces ameliorated generalization ability on shifted data distributions such as noisy or perturbed data.
  Summary In this section, we examined the structured sparsity, noise robustness, and trasferability of the winning tickets. Throughout the studies, we found that winning tickets can generalize to structured sparsity with no performance degradation. In addition, winning tickets identified from source datasets can achieve matching performance on target datasets, which verifies the transferability. Lastly, winning tickets (identified from either target dataset or source datasets) are significantly more robust to noise compared to full models, especially when the noise level is high. These results jointly demonstrate the eligibility and benefits of winning tickets in on-device ASR applications.
sections.heading: 6 CONCLUSIONS AND FUTURE WORKS.
sections.text: In this work, we examine and leverage lottery ticket hypothesis in speech recognition for the first time. Our extensive results show that the winning tickets are not only ultra-lightweight, but also highly transferable and (even more) noise-robust, compared to the full models. These results collectively propound the use of LTH into ASR models, bringing new insights to both LTH theory and portable ASR research. The future works to this study is to evaluate the actual hardware benefits of our found winning tickets. For example, Block-Sparse GPU Kernels (Gray et al., 2017) can achieve over 5x speed up with over 80% block sparse network, and XNNPACK (Elsen et al., 2020) can speed up networks at 70%-90% unstructured sparsity on smartphone processors, which applies to our found winning tickets. In addition, we also would like to generalize LTH compression paradigm to other speech tasks such as text-to-speech synthesis (Wang et al., 2017; Shen et al., 2018) and voice-to-voice conversion (Jia et al., 2019; Biadsy et al., 2019; Zhao et al., 2019). These new directions require further customized studies and point to new opportunities for LTH in speech research.
sections.heading: 7 REPRODUCIBILITY STATEMENT
sections.text: To ensure reproducibility, we the Machine Learning Reproducibility Checklist v2.0, Apr. 7 2020 (Pineau et al., 2021).
  • For all models and algorithms presented,
  – A clear description of the mathematical settings, algorithm, and/or model. We clearly describe all of the settings, formulations, and algorithms in Section 3.
  – A clear explanation of any assumptions. We do not make assumptions. – An analysis of the complexity (time, space, sample size) of any algorithm. We use
  the number of parameters as the spatial complexity of our models, as shown in Table 1.
  • For any theoretical claim,
  – A clear statement of the claim. We do not make theoretical claims. – A complete proof of the claim. We do not make theoretical claims.
  • For all datasets used, check if you include:
  – The relevant statistics, such as number of examples. We used publicly available speech corpora in all of our experiments, and we cite the papers of them following the authors instructions, and we provide related statistics in Section 3.
  – The details of train/validation/test splits. We use the default train/validation/test splits of the three public datasets, as described in Section 3.
  – An explanation of any data that were excluded, and all pre-processing step. We did not exclude any data. Our pre-processing steps are explained in A.1.
  – A link to a downloadable version of the dataset or simulation environment. All the datasets are publicly available, and their downloadable links can be easily found at their original papers that we cite in Section 3.
  – For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control. We do not collect or release new datasets.
  • For all shared code related to this work, check if you include:
  – All of our code will be released after the paper was peer-reviewed, as we mentioned in Abstract. We also include the key frameworks and libraries that we used during implementation in Appendix A.2.
  • For all reported experimental results, check if you include:
  – The range of hyper-parameters considered, method to select the best hyperparameter configuration, and specification of all hyper-parameters used to generate results. We provide all details of the hyper-parameter we used in in Section 3 and Appendix A.1. We did not perform any hyper-parameter tuning in this work since the datasets we used are with large scale, and performing hyper-parameter tuning on large scale datasets are extremely time- and resource-consuming.
  – The exact number of training and evaluation runs. One run is conducted for each experiments.
  – A clear definition of the specific measure or statistics used to report results. We use the word error rate (WER) as the performance metric and the remaining weights (RW) as the complexity metric on test sets,
  – A description of results with central tendency (e.g. mean) & variation (e.g. error bars). We do not report error bars.
  – The average runtime for each result, or estimated energy cost. We do not report the running time or energy cost.
  – A description of the computing infrastructure used. All detailed descriptions are presented in Section 3 and Appendix A.1.
sections.heading: A APPENDIX
sections.heading: A.1 DETAILS IN BACKBONE ARCHITECTURES
sections.text: Data pre-processing. The input to all the backbone networks is a sequence of 161-dimensional magnitude spectrogram (160 magnitude points plus 1 DC component), extracted with 20ms window, 10ms shift, and 320-point fast fourier transform. The ground-truth labels are represented by 28 graphemes, including 26 English characters, space, and apostrophe symbol (an extra blank token “-” is also used for CTC backbone).
  CTC. The network is composed of two convolutional layers and five recurrent layers, followed by a final fully-connected layer. The first convolutional layer has 32 41×11 (in frequency×time) kernels, with 2×2 stride. The second convolutional layer has 32 21×11 kernels, with 2×1 stride. For each convlutional layers, tanh is used as nonlinearity. Following these, there are five bi-directional RNN layers, each of them having 1,024 units. A final fully-connected layer with softmax operator outputs a probability distribution over characters. The total number of parameters in the full model is 86.62M.
  RNN-Transducer. In RNN-Transducer, we set the encoder and decoder to have 5 and 1 bidirectionalLSTM layers, each of them having 1,024 units. The total number of parameters in the full model is 132.23M.
  Conformer. In Conformer, we used the same hyper-parameters as the Conformer(L) in [4]. The model has 17 encoder layers, each of them is composed of 8-head convolution augmented attention. The convolution kernel has a shape of 1×32, and the fully-connected layers in attention is 512- dimensional. The decoder layer is the same as RNN-Transducer, with 1,024 units. The total number of parameters in the full model is 118.40M.
sections.heading: A.2 MORE IMPLEMENTATIONAL DETAILS
sections.text: As we mentioned in abstract, we will open-source all of the code once the paper is peer-reveiwed. Here, we provide key references and tools that we used during our implementation.
  We implement neural network training using PyTorch library (Paszke et al., 2019). Our implementation of CTC backbone is based on:
  • https://github.com/SeanNaren/deepspeech.pytorch
  Our implementation of RNN-Transduce and Conformer is based on:
  • https://github.com/openspeech-team/openspeech
  We used PyTorch pruning library for unstructured sparsity:
  • https://pytorch.org/docs/stable/generated/torch.nn.utils. prune.global_unstructured.html
  We implemented pruning functions for structured sparsity based on:
  • https://pytorch.org/docs/stable/generated/torch.nn.utils. prune.RandomStructured.html
  • https://github.com/openai/blocksparse
  • https://www.tensorflow.org/model_optimization/guide/pruning
sections.heading: A.3 EVALUATING WINNING TICKETS OF THREE ASR BACKBONES ON LIBRISPEECH TEST-OTHER SUBSET
sections.text: The performance of the three backbones on LibriSpeech test-other subset is shown in Table 7.
sections.heading: A.4 MORE EXAMPLES OF THE MODEL OUTPUTS
sections.text: We provided a couple more examples of the outputs from full model, winning tickets at extreme sparsity, and best performing winning tickets on TED-LIUM test utterances in Figure 6. The examples in the first row show the cases where the winning tickets avoid the errors appeared in the full model outputs. Meanwhile, we also noticed a few cases where winning tickets produce erroneous recognition results, as shown in the second and the third rows. Most of these utterances have either a very fast speaking rate or unclear pronunciations of words. Therefore, neither full model nor winning tickets reasonably transcribe the utterances in challenging cases, showing the limited capability of the models.
sections.heading: A.5 COMPARISON TO OTHER MODEL COMPRESSION APPROACHES ON LIBRISPEECH TEST-OTHER SUBSET
sections.text: Comparison between the proposed approach and state-of-the-art distillation and pruning methods on LibriSpeech test-other subset is shown in Table 8. Sequence-level KD approach (Takashima et al., 2018) does not report their evaluation results on test-other subset, and therefore, we use “N/A” in the table.
sections.heading: A.6 MORE RESULTS IN TRANSFERRING STUDY
sections.text: We have shown the WER curves of the transferring winning tickets transferred in Figure 5. Here, we included the WER and RW at the extreme sparsity in Table 9, Table10 and Table 11, respectively. The transferring tickets to both TED-LIUM and CommonVoice have a extreme remaining weights of 32.8%, and those to LibriSpeech have a extreme remaining weights of 32.8% compared to the full models. Although they are not as sparse as the winning tickets identified from the target datasets, the transferring tickets are still much more efficient than full models. These results are in consensus with our findings in Section 5.
sections.heading: A.7.2 RUN TIME EVALUATIONS
sections.heading: A.7.1 RESULTS ON THE EXISTENCE OF THE THREE BACKBONES ON TED-LIUM AND COMMONVOICE
sections.heading: A.7 EXTRA NEW RESULTS
sections.heading: A.7.3 UPDATED NOISE ROBUSTNESS RESULTS
sections.heading: A.7.4 BLOCK SPARSITY EXPERIMENTS WITH LARGER BLOCK SIZE
sections.heading: A.7.5 PSEUDO-CODE OF LTH PRUNING ALGORITHM
sections.text: Algorithm 1 Lottery Ticket Hypothesis Pruning 1: Set the initial mask m, with the weight initialization θ. 2: repeat 3: Rewind the weight to θ 4: Train f(x;m θ) for t epochs with algorithm ADt , i.e., ADt (f(x;m θ)) 5: Prune 20% of remaining weights in ADt (f(x;m θ)) and update m accordingly 6: until the sparsity of m reaches the desired sparsity level s 7: Return f(x;m θ).
sections.heading: A.7.6 FURTHER EXPLANATION OF BLOCK SPARSITY.
sections.text: In terms of block sparsity, it prunes a block of parameters (e.g., 1x4 block) instead of just one parameter, which has the minimal magnitude. Unstructured pruning could be thought as block sparsity with 1x1 block.
sections.heading: A.7.7 EXPLANATION OF THE CHOICE OF SPARSITY LEVELS
sections.text: In our current setting, we prune out the 20% of remaining weights that have the lowest magnitude. The reason for choosing the value to be 20% is based on considerations of model performance and computation resources. When the amount of weights pruned at each iteration becomes larger, less iterations will be needed. However, it is more likely to have larger performance regressions, since the pruned weights could be useful in the subnetwork at higher sparsity. On the other hand, if the amount of weight pruned at each iteration is too small, the number of iterations required would be very large,
  which is extremely resource demanding. As a result, we feel 20% is a good one considering both factors.
title: AUDIO LOTTERY: SPEECH RECOGNITION MADE