summaries: This paper conducts comprehensive analyses of lottery tickets hypothesis on automatic speech recognition. The authors verified the existence of highly sparse “winning tickets” in ASR task, and analyze its robustness to noise, transferable to other datasets, and supports with structured sparsity. They conduct experiments on different model structures (CTC, RNN-T, Transfomer) and different datasets (LibriSpeech, TED-LIUM, Common Voice) with extensive analyses.

summaries: In this paper, authors propose to use lottery ticket hypothesis (LTH) for ASR model pruning. The whole idea generally inherits from the original LTH paper. The model is first trained from scratch. Then you collect all the model weights and sort them. A proportion of the smallest weights are set and fixed to 0. Given this sparsified subnetwork, one further repeats the whole process and prune more parameters. After several rounds, if the pruned subnetwork still gives at least the same performance of the original full network , it is called a "winning ticket". The "winning ticket" subnetwork should be lightweight, transferrable, and noise-robust. This work applies LTH to 3 models (CNN-RNN CTC, RNN-T, Conformer) on 3 datasets (TED-LIUM, Common Voice, LibriSpeech). The authors empirically show  1. The winning ticket exists for each method. The subnetwork could be only 4.4% size of the original full model, but still matches the performance. 2. Compared to other pruning methods, LTH performs best. 3. LTH with pretrained model performs better than LTH with random initialization.  Furthermore, the paper shows 1. One can enforce block sparsity for pruning to facilitate chip design. 2. The winning ticket can transfer to a new dataset and work well. 3. The pruned network is noise-robust.

summaries: This paper investigates using the lottery tickets hypothesis (LTH) strategy for pruning neural network weights for speech recognition. The method first explains the general LTH framework and extensions with transfer learning scenarios. The paper shows the effectiveness of the proposed method in the standard ASR tasks, pre-training with other models, and transfer learning, especially in noisy speech recognition tasks.  Some comments: - In the introduction "End-to-end automatic speech recognition (ASR) (Wang et al., 2019)": (Wang et al., 2019) is not a representative paper of end-to-end ASR - In the introduction, "For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020)": Again not sure (Kinoshita et al., 2020) is a correct citation. I think the following papers are more appropriate   - Haeb-Umbach, Reinhold, et al. "Speech processing for digital home assistants: Combining signal processing with deep-learning techniques." IEEE Signal processing magazine 36.6 (2019): 111-124.   - Kinoshita, Keisuke, et al. "The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech." 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. IEEE, 2013. - Currently, data augmentation techniques (artificially contaminate the clean speech with various noisy data) are more standard than transfer learning for noisy speech recognition. Please discuss it. - Section 2 "On the model level, compared to CV models, speech models are mostly based on RNN backbones": I'm confused about this description. As you mentioned in the previous paragraphs, many speech models now use transformer, conformer, and CNN. - Section 2 "with 10ms shift is 1,000": this is not exactly true as we usually use downsampling at the beginning of the encoder layer. - Section 3, RQ3: Frankly, I don't think this is an essential question for me. This sounds very trivial that the use of pre-trained models would improve the performance. - Section 4, What is the main message of Figure 1? I don't find the particular effectiveness of the proposed method from this figure/example. - Section 4, Table 3: Why are the other methods, especially KD-based methods, so worse? - Section 5 "A common way to address this issue is through speaker adaptation (Zhao, 1994;": Again, I don't think (Zhao, 1994) is a representative paper for speaker adaptation. MAP/MLLR adaptation techniques exist in this era, and they were standard techniques in the GMM era. - Section 5: I recommend the authors use the real noisy speech data like the CHiME data

summaries: The paper explores the lottery ticket hypothesis, i.e., the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models within the context of ASR. The authors show that it is possible to find subnetworks which contain just a fraction of the number of parameters of the full models without sacrificing performance on ASR. Three different architectures are considered and results reported on 3 datasets. Additional studies investigating the transferability and robustness of the sparse subnetworks are also conducted.

abstractText: Lightweight speech recognition models have seen explosive demands owing to a growing amount of speech-interactive features on mobile devices. Since designing such systems from scratch is non-trivial, practitioners typically choose to compress large (pre-trained) speech models. Recently, lottery ticket hypothesis reveals the existence of highly sparse subnetworks that can be trained in isolation without sacrificing the performance of the full models. In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to fit the open-world personalization; and 3) compatible with structured sparsity. We conducted extensive experiments on CTC, RNN-Transducer, and Transformer models, and verified the existence of highly sparse “winning tickets” that can match the full model performance across those backbones. We obtained winning tickets that have less than 20% of full model weights on all backbones, while the most lightweight one only keeps 4.4% weights. Those winning tickets generalize to structured sparsity with no performance loss, and transfer exceptionally from large source datasets to various target datasets. Perhaps most surprisingly, when the training utterances have high background noises, the winning tickets even substantially outperform the full models, showing the extra bonus of noise robustness by inducing sparsity. All our codes will be released.

authors:
Dario Amodei
Sundaram Ananthanarayanan
Rishita Anubhai
Jingliang Bai
Eric Battenberg
Carl Case
Jared Casper
Bryan Catanzaro
Qiang Cheng
Guoliang Chen

title: Deep speech 2: End-to-end speech recognition in english and mandarin

venue: In ICML,

authors:
Rosana Ardila
Megan Branson
Kelly Davis
Michael Kohler
Josh Meyer
Michael Henretty
Reuben Morais
Lindsay Saunders
Francis Tyers
Gregor Weber

title: Common Voice: A massivelymultilingual speech corpus

venue: In Proceedings of The 12th Language Resources and Evaluation Conference,

authors:
Sercan Ö Arık
Markus Kliegl
Rewon Child
Joel Hestness
Andrew Gibiansky
Chris Fougner
Ryan Prenger
Adam Coates

title: Convolutional recurrent neural networks for small-footprint keyword spotting

venue: In INTERSPEECH,

authors:
Alexei Baevski
Henry Zhou
Abdelrahman Mohamed
Michael Auli

title: wav2vec 2.0: A framework for self-supervised learning of speech representations

venue: arXiv preprint arXiv:2006.11477,

authors:
Dzmitry Bahdanau
Jan Chorowski
Dmitriy Serdyuk
Philemon Brakel
Yoshua Bengio

title: Endto-end attention-based large vocabulary speech recognition

venue: In ICASSP,

authors:
Fadi Biadsy
Ron J Weiss
Pedro J Moreno
Dimitri Kanvesky
Ye Jia

title: Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation

authors:
Davis Blalock
Jose Javier Gonzalez Ortiz
Jonathan Frankle
John Guttag

title: What is the state of neural network pruning

venue: arXiv preprint arXiv:2003.03033,

authors:
William Chan
Navdeep Jaitly
Quoc Le
Oriol Vinyals

title: Listen, attend and spell: A neural network for large vocabulary conversational speech recognition

authors:
Tianlong Chen
Jonathan Frankle
Shiyu Chang
Sijia Liu
Yang Zhang
Michael Carbin
Zhangyang Wang

title: The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models

venue: arXiv preprint arXiv:2012.06908,

authors:
Tianlong Chen
Jonathan Frankle
Shiyu Chang
Sijia Liu
Yang Zhang
Zhangyang Wang
Michael Carbin

title: The lottery ticket hypothesis for pre-trained BERT networks

venue: NeurIPS,

authors:
Tianlong Chen
Yu Cheng
Zhe Gan
Jingjing Liu
Zhangyang Wang

title: Ultra-data-efficient GAN training: Drawing a lottery ticket first, then training it toughly

venue: arXiv preprint arXiv:2103.00397,

authors:
Tianlong Chen
Yongduo Sui
Xuxi Chen
Aston Zhang
Zhangyang Wang

title: A unified lottery ticket hypothesis for graph neural networks

venue: arXiv preprint arXiv:2102.06790,

authors:
Tianlong Chen
Zhenyu Zhang
Sijia Liu
Shiyu Chang
Zhangyang Wang

title: Long live the lottery: The existence of winning tickets in lifelong learning

venue: In ICLR,

authors:
Xiaohan Chen
Yu Cheng
Shuohang Wang
Zhe Gan
Zhangyang Wang
Jingjing Liu

title: Earlybert: Efficient bert training via early-bird lottery tickets

venue: arXiv preprint arXiv:2101.00063,

authors:
Xuxi Chen
Zhenyu Zhang
Yongduo Sui
Tianlong Chen

title: GANs can play lottery tickets too

authors:
Chung-Cheng Chiu
Tara N Sainath
Yonghui Wu
Rohit Prabhavalkar
Patrick Nguyen
Zhifeng Chen
Anjuli Kannan
Ron J Weiss
Kanishka Rao
Ekaterina Gonina

title: State-of-the-art speech recognition with sequence-to-sequence models

authors:
Jan Chorowski
Dzmitry Bahdanau
Dmitriy Serdyuk
Kyunghyun Cho
Yoshua Bengio

title: Attention-based models for speech recognition

venue: In ICONIP, pp

title: Embedded speech recognition applications in mobile phones: Status, trends, and challenges

authors:
Shrey Desai
Hongyuan Zhan
Ahmed Aly

title: Evaluating lottery tickets under distributional shifts

venue: arXiv preprint arXiv:1910.12708,

authors:
Fenglin Ding
Wu Guo
Bin Gu
Zhenhua Ling
Jun Du

title: Adaptive speaker normalization for ctc-based speech recognition

authors:
Linhao Dong
Shuang Xu
Bo Xu

title: Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Linhao Dong
Shiyu Zhou
Wei Chen
Bo Xu

title: Extending recurrent neural aligner for streaming end-to-end speech recognition in Mandarin

authors:
Erich Elsen
Marat Dukhan
Trevor Gale
Karen Simonyan

title: Fast sparse convnets

venue: In CVPR,

authors:
Utku Evci
Fabian Pedregosa
Aidan Gomez
Erich Elsen

title: The difficulty of training sparse neural networks

venue: arXiv preprint arXiv:1906.10732,

authors:
Florian Eyben
Martin Wöllmer
Björn Schuller
Alex Graves

title: From speech to letters-using a novel neural network architecture for grapheme based ASR

venue: IEEE Workshop on Automatic Speech Recognition & Understanding,

authors:
Jonathan Frankle
Michael Carbin

title: The lottery ticket hypothesis: Finding sparse, trainable neural networks

venue: In ICLR, 2019a. URL https://openreview.net/forum?id=rJl-b3RcF7

authors:
Jonathan Frankle
Gintare Karolina Dziugaite
Daniel Roy
Michael Carbin

title: Linear mode connectivity and the lottery ticket hypothesis

authors:
Jonathan Frankle
David J. Schwab
Ari S. Morcos

title: The early phase of neural network training

venue: In ICLR, 2020c. URL https://openreview.net/forum?id=Hkl1iRNFwS

authors:
Trevor Gale
Erich Elsen
Sara Hooker

title: The state of sparsity in deep neural networks

venue: arXiv preprint arXiv:1902.09574,

authors:
Dawei Gao
Xiaoxi He
Zimu Zhou
Yongxin Tong
Ke Xu
Lothar Thiele

title: Rethinking pruning for accelerating deep inference at the edge

venue: In SIGKDD,

authors:
J-L Gauvain
Chin-Hui Lee

title: Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains

venue: IEEE transactions on speech and audio processing,

authors:
Sharath Girish
Shishira R Maiya
Kamal Gupta
Hao Chen
Larry Davis
Abhinav Shrivastava

title: The lottery ticket hypothesis for object recognition

venue: arXiv preprint arXiv:2012.04643,

title: Sequence transduction with recurrent neural networks

venue: arXiv preprint arXiv:1211.3711,

authors:
Alex Graves
Navdeep Jaitly

title: Towards end-to-end speech recognition with recurrent neural networks

authors:
Alex Graves
Santiago Fernández
Faustino Gomez
Jürgen Schmidhuber

title: Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks

venue: In ICML, pp

authors:
Alex Graves
Abdel-rahman Mohamed
Geoffrey Hinton

title: Speech recognition with deep recurrent neural networks

authors:
Scott Gray
Alec Radford
Diederik P Kingma

title: Gpu kernels for block-sparse weights

venue: arXiv preprint arXiv:1711.09224,

authors:
Shupeng Gui
Haotao Wang
Chen Yu
Haichuan Yang
Zhangyang Wang
Ji Liu

title: Model compression with adversarial robustness: A unified optimization framework

authors:
Anmol Gulati
James Qin
Chung-Cheng Chiu
Niki Parmar
Yu Zhang
Jiahui Yu
Wei Han
Shibo Wang
Zhengdong Zhang
Yonghui Wu

title: Conformer: Convolution-augmented transformer for speech recognition

authors:
Reinhold Haeb-Umbach
Shinji Watanabe
Tomohiro Nakatani
Michiel Bacchiani
Bjorn Hoffmeister
Michael L Seltzer
Heiga Zen
Mehrez Souden

venue: IEEE Signal processing magazine,

authors:
Song Han
Huizi Mao
William J Dally

title: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

venue: arXiv preprint arXiv:1510.00149,

authors:
Awni Hannun
Carl Case
Jared Casper
Bryan Catanzaro
Greg Diamos
Erich Elsen
Ryan Prenger
Sanjeev Satheesh
Shubho Sengupta
Adam Coates

title: Deep speech: Scaling up end-to-end speech recognition

venue: arXiv preprint arXiv:1412.5567,

authors:
Yanzhang He
Tara N Sainath
Rohit Prabhavalkar
Ian McGraw
Raziel Alvarez
Ding Zhao
David Rybach
Anjuli Kannan
Yonghui Wu
Ruoming Pang

title: Streaming end-to-end speech recognition for mobile devices

authors:
Navdeep Jaitly
Patrick Nguyen
Andrew Senior
Vincent Vanhoucke

title: Application of pretrained deep neural networks to large vocabulary speech recognition

authors:
Ye Jia
Ron J Weiss
Fadi Biadsy
Wolfgang Macherey
Melvin Johnson
Zhifeng Chen
Yonghui Wu

title: Direct speech-to-speech translation with a sequence-to-sequence model

authors:
Neha Mukund Kalibhat
Yogesh Balaji
Soheil Feizi

title: Winning lottery tickets in deep generative models

venue: arXiv preprint arXiv:2010.02350,

authors:
Kwangyoun Kim
Kyungmin Lee
Dhananjaya Gowda
Junmo Park
Sungsoo Kim
Sichen Jin
Young-Yoon Lee
Jinsu Yeo
Daehyun Kim
Seokyeong Jung

title: Attention based on-device streaming speech recognition with large speech corpus

venue: In ASRU,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization, 2017

authors:
Keisuke Kinoshita
Marc Delcroix
Takuya Yoshioka
Tomohiro Nakatani
Emanuel Habets
Reinhold Haeb-Umbach
Volker Leutnant
Armin Sehr
Walter Kellermann
Roland Maas

title: The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech

venue: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics,

authors:
Keisuke Kinoshita
Tsubasa Ochiai
Marc Delcroix
Tomohiro Nakatani

title: Improving noise robust automatic speech recognition with single-channel time-domain enhancement network

authors:
Markus Kliegl
Siddharth Goyal
Kexin Zhao
Kavya Srinet
Mohammad Shoeybi

title: Trace norm regularization and faster inference for embedded speech recognition RNNs

venue: arXiv preprint arXiv:1710.09026,

authors:
Christopher J Leggetter
Philip C Woodland

title: Maximum likelihood linear regression for speaker adaptation of continuous density hidden markov models

venue: Computer speech & language,

authors:
Chenxing Li
Lei Zhu
Shuang Xu
Peng Gao
Bo Xu

title: Compression of acoustic model via knowledge distillation and pruning

authors:
Jinyu Li
Li Deng
Yifan Gong
Reinhold Haeb-Umbach

title: An overview of noise-robust automatic speech recognition

venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,

authors:
Ke Li
Jinyu Li
Yong Zhao
Kshitiz Kumar
Yifan Gong

title: Speaker adaptation for end-to-end CTC models

venue: In SLT,

authors:
Zerui Li
Yue Ming
Lei Yang
Jing-Hao Xue

title: Mutual-learning sequence-level knowledge distillation for automatic speech

venue: recognition. Neurocomputing,

authors:
Zhuohan Li
Eric Wallace
Sheng Shen
Kevin Lin
Kurt Keutzer
Dan Klein
Joey Gonzalez

title: Train big, then compress: Rethinking model size for efficient training and inference of transformers

title: Speaker adaptation of context dependent deep neural networks

authors:
Hong Liu
Mingsheng Long
Jianmin Wang
Michael I Jordan

title: Towards understanding the transferability of deep representations

venue: arXiv preprint arXiv:1909.12031,

authors:
Zhuang Liu
Mingjie Sun
Tinghui Zhou
Gao Huang
Trevor Darrell

title: Rethinking the value of network pruning

authors:
Haoyu Ma
Tianlong Chen
Ting-Kuei Hu
Chenyu You
Xiaohui Xie
Zhangyang Wang

title: Good students play big lottery better

venue: arXiv preprint arXiv:2101.03255,

authors:
Ian McGraw
Rohit Prabhavalkar
Raziel Alvarez
Montse Gonzalez Arenas
Kanishka Rao
David Rybach
Ouais Alsharif
Haşim Sak
Alexander Gruenstein
Françoise Beaufays

title: Personalized speech recognition on mobile devices

title: Sparse transfer learning via winning lottery tickets

venue: arXiv preprint arXiv:1905.07785,

authors:
Zhong Meng
Yashesh Gaur
Jinyu Li
Yifan Gong

title: Speaker adaptation for attention-based end-to-end speech recognition

authors:
Yajie Miao
Mohammad Gowayyed
Florian Metze

title: EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding

authors:
Ari Morcos
Haonan Yu
Michela Paganini
Yuandong Tian

title: One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers

venue: In NeurIPS,

authors:
Sharan Narang
Erich Elsen
Gregory Diamos
Shubho Sengupta

title: Exploring sparsity in recurrent neural networks

venue: arXiv preprint arXiv:1704.05119,

authors:
Sharan Narang
Eric Undersander
Gregory Diamos

title: Block-sparse recurrent neural networks

venue: arXiv preprint arXiv:1711.02782,

authors:
Vassil Panayotov
Guoguo Chen
Daniel Povey
Sanjeev Khudanpur

title: Librispeech: An ASR corpus based on public domain audio books

authors:
Jinhwan Park
Yoonho Boo
Iksoo Choi
Sungho Shin
Wonyong Sung

title: Fully neural network based speech recognition on mobile and embedded devices

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

title: Pytorch: An imperative style, high-performance deep learning

venue: library. NeurIPS,

authors:
Joelle Pineau
Philippe Vincent-Lamarre
Koustuv Sinha
Vincent Lariviere
Alina Beygelzimer
Florence d’Alché Buc
Emily Fox
Hugo Larochelle

title: Improving reproducibility in machine learning research

venue: Journal of Machine Learning Research,

authors:
Rohit Prabhavalkar
Tara N Sainath
Yonghui Wu
Patrick Nguyen
Zhifeng Chen
Chung-Cheng Chiu
Anjuli Kannan

title: Minimum word error rate training for attention-based sequence-to-sequence models

authors:
Sai Prasanna
Anna Rogers
Anna Rumshisky

title: When BERT plays the lottery, all tickets are winning

venue: arXiv preprint arXiv:2005.00561,

authors:
Kanishka Rao
Haşim Sak
Rohit Prabhavalkar

title: Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-Transducer

authors:
Alex Renda
Jonathan Frankle
Michael Carbin

title: Comparing rewinding and fine-tuning in neural network pruning

authors:
Anthony Rousseau
Paul Deléglise
Yannick Esteve

title: TED-LIUM: an automatic speech recognition dedicated corpus

venue: In LREC, pp

authors:
Tara N Sainath
Yanzhang He
Bo Li
Arun Narayanan
Ruoming Pang
Antoine Bruguier
Shuo-yiin Chang
Wei Li
Raziel Alvarez
Zhifeng Chen

title: A streaming on-device end-to-end model surpassing server-side conventional model quality and latency

authors:
Pedro Savarese
Hugo Silva
Michael Maire

title: Winning the lottery with continuous sparsification

authors:
Johan Schalkwyk
Doug Beeferman
Françoise Beaufays
Bill Byrne
Ciprian Chelba
Mike Cohen
Maryam Kamvar
Brian Strope

title: Your word is my command”: Google search by voice: A case study

venue: In Advances in speech recognition,

authors:
Romain Serizel
Nicolas Turpault
Ankit Shah
Justin Salamon

title: Sound event detection in synthetic domestic environments

authors:
Yuan Shangguan
Jian Li
Qiao Liang
Raziel Alvarez
Ian McGraw

title: Optimizing speech recognition for the edge

venue: arXiv preprint arXiv:1909.12408,

authors:
Jonathan Shen
Ruoming Pang
Ron J Weiss
Mike Schuster
Navdeep Jaitly
Zongheng Yang
Zhifeng Chen
Yu Zhang
Yuxuan Wang
Rj Skerrv-Ryan

title: Natural TTS synthesis by conditioning Wavenet on Mel spectrogram predictions

authors:
Khe Chai Sim
Petr Zadrazil
Françoise Beaufays

title: An investigation into on-device personalization of end-to-end automatic speech recognition models

authors:
Ryoichi Takashima
Sheng Li
Hisashi Kawai

title: An investigation of a knowledge distillation method for ctc acoustic models

authors:
Ryu Takeda
Kazuhiro Nakadai
Kazunori Komatani

title: Node pruning based on entropy of weights and node activity for small-footprint acoustic model based on deep neural networks

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: In Advances in neural information processing systems,

authors:
Alex Waibel
Ahmed Badran
Alan W Black
Robert Frederking
Donna Gates
Alon Lavie
Lori Levin
Kevin A Lenzo
Laura Mayfield Tomokiyo
Jurgen Reichert

title: Speechalator: Two-way speech-to-speech translation on a consumer PDA

venue: In Eighth European Conference on Speech Communication and Technology,

authors:
Chaoqi Wang
Guodong Zhang
Roger Grosse

title: Picking winning tickets before training by preserving gradient flow

authors:
Dong Wang
Xiaodong Wang
Shaohe Lv

title: An overview of end-to-end automatic speech recognition. Symmetry

authors:
Yongqiang Wang
Abdelrahman Mohamed
Due Le
Chunxi Liu
Alex Xiao
Jay Mahadeokar
Hongzhao Huang
Andros Tjandra
Xiaohui Zhang
Frank Zhang

title: Transformer-based acoustic modeling for hybrid speech recognition

authors:
Yuxuan Wang
RJ Skerry-Ryan
Daisy Stanton
Yonghui Wu
Ron J Weiss
Navdeep Jaitly
Zongheng Yang
Ying Xiao
Zhifeng Chen
Samy Bengio

title: Tacotron: Towards end-to-end speech synthesis

authors:
Wei Wen
Yuxiong He
Samyam Rajbhandari
Minjia Zhang
Wenhan Wang
Fang Liu
Bin Hu
Yiran Chen
Hai Li

title: Learning intrinsic sparse structures within long short-term memory

authors:
Felix Weninger
Jesús Andrés-Ferrer
Xinwei Li
Puming Zhan

title: Listen, attend, spell and adapt: Speaker adapted sequence-to-sequence ASR

title: Speaker adaptation for continuous density hmms: A review

venue: In ISCA Tutorial and Research Workshop (ITRW) on Adaptation Methods for Speech Recognition,

authors:
Zhaofeng Wu
Ding Zhao
Qiao Liang
Jiahui Yu
Anmol Gulati
Ruoming Pang

title: Dynamic sparsity neural networks for automatic speech recognition

authors:
Shaokai Ye
Kaidi Xu
Sijia Liu
Hao Cheng
Jan-Henrik Lambrechts
Huan Zhang
Aojun Zhou
Kaisheng Ma
Yanzhi Wang
Xue Lin

title: Adversarial robustness vs. model compression, or both

authors:
Ji Won Yoon
Hyeonseung Lee
Hyung Yong Kim
Won Ik Cho
Nam Soo Kim

title: Tutornet: Towards flexible knowledge distillation for end-to-end speech recognition

authors:
Haoran You
Chaojian Li
Pengfei Xu
Yonggan Fu
Yue Wang
Xiaohan Chen
Richard G. Baraniuk
Zhangyang Wang
Yingyan Lin

title: Drawing early-bird tickets: Toward more efficient training of deep networks

authors:
Haonan Yu
Sergey Edunov
Yuandong Tian
Ari S Morcos

title: Playing the lottery with rewards and multiple languages: Lottery tickets in RL and NLP

authors:
Heiga Zen
Viet Dang
Rob Clark
Yu Zhang
Ron J Weiss
Ye Jia
Zhifeng Chen
Yonghui Wu

title: LibriTTS: A corpus derived from LibriSpeech for Text-to-Speech

authors:
Shunshi Zhang
Bradly C Stadie

title: One-shot pruning of recurrent neural networks by jacobian spectrum evaluation

authors:
Yu Zhang
William Chan
Navdeep Jaitly

title: Very deep convolutional networks for end-to-end speech recognition

authors:
Guanlong Zhao
Shaojin Ding
Ricardo Gutierrez-Osuna

title: Foreign accent conversion by synthesizing speech from phonetic posteriorgrams

heading: 1 INTRODUCTION

text:
End-to-end automatic speech recognition (ASR) (Wang et al., 2019; Hannun et al., 2014; Graves, 2012; Chorowski et al., 2015; Dong et al., 2018a) has become an indispensable technology in consumer-interactive devices (e.g., smartphones, smart speakers, tablets) over the past few years (He et al., 2019; Cohen, 2008; Schalkwyk et al., 2010). Conventional on-device ASR system usually requires the involvement of servers, i.e., streaming the audio to servers and then streaming the results back to the devices. By contrast, recent studies (He et al., 2019; McGraw et al., 2016; Kim et al., 2019; Sim et al., 2019; Park et al., 2018; Waibel et al., 2003; Arık et al., 2017) have spurred the success of ASR models fully run on devices, which can be advantageous in terms of computational resources, latency, and user data privacy.
Developing on-device ASR models are challenging since the computational resources (e.g., CPU, memory, battery) are typically very limited. A standard design approach to fit ASR model in budget is through applying various neural network compression techniques to the larger ASR models, such as network pruning (Takeda et al., 2017; Shangguan et al., 2019; Gao et al., 2020), knowledge distillation (Li et al., 2018a), and parameter quantization (He et al., 2019; Sainath et al., 2020). However, there always exists a trade-off between computational efficiency and the model performance, and the efficiency improvements are usually at the cost of word error rate (WER). In most prior works, we observed a non-negligible degradation of WER in compressed models.
A new horizon emerges with the discovery of lottery tickets hypothesis (LTH) (Frankle & Carbin, 2019b). LTH empirically demonstrated the existence of highly sparse matching subnetworks (i.e., winning tickets) in full dense networks, that can be independently trained from scratch to match or even surpass the performance of the latter. LTH was widely observed in various models in computer vision (Liu et al., 2019b; Evci et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al.,
2020a; You et al., 2020; Ma et al., 2021; Girish et al., 2020) and natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna et al., 2020; Chen et al., 2020b;c). Yet to our best knowledge, it has not been studied nor utilized in the realm of speech processing and recognition .
This paper presents the first investigation on LTH for developing on-device ASR models. Despite the rich literature of LTH in vision and language, a practically useful winning ticket for real-world ASR would demand two unique properties: transferability, noise robustness. • As one persistent challenges of ASR, each individual has a different voice and speaking style.
Unlike text or images, whose data are much more “standardized”, the spoken word varies greatly based on regional dialects, speed, emphasis, even social class and gender. Therefore, scaling up any ASR system has always been a significant obstacle, since the testing utterances may have very different distribution from the training utterances. That has made transferability a crucial demand for ASR in the open world.
• In an ideal world, one would have to speak very clearly, slowly, and in an environment with no background noise, for the sounds being recognized: that unfortunately will not happen in the practice. In the real-world ASR applications, noise robustness is becoming another crucially demanded technological factor since ASR is now expected to work in much more difficult acoustic environments than in the past (Li et al., 2014). For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020; Haeb-Umbach et al., 2019; Kinoshita et al., 2013). Even in the standard ASR benchmark such as LibriSpeech (Panayotov et al., 2015), there are significant background noise in its “clean” subset (Zen et al., 2019).
More importantly, prior LTH studies mostly use unstructured sparsity during model pruning. However, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU Kernels (Gray et al., 2017)). Together with the two unique gaps, they make ASR no less challenging than visual recognition or text understanding, if not more. They account for the prior arts’ difficulty to maintain a satisfactory balance between model efficiency and recognition performance; and similarly, they question the applicability of LTH in ASR.
We provide an affirmative, positive answer of LTH in ASR. As the subject of study, we choose most commonly used ASR architectures in both research and products over the past few years: 1) CNN+RNN with connectionist temporal classification (CTC) (Graves et al., 2006); 2) RNNTransducer (Graves, 2012); and 3) Convolution-augmented Transformer (Gulati et al., 2020). We conducted extensive LTH experiments using these backbones on three popular corpora: TEDLIUM (Rousseau et al., 2012), Common Voice (Ardila et al., 2020), and LibriSpeech (Panayotov et al., 2015). Unlike most of existing LTH studies that only approached to explaining and demonstrating the correctness of LTH theory, in this work, we make the first attempt to apply LTH to real-world use cases. Namely, we investigate three unique properties that were rarely studied in previous LTH research but are key to user-interactive ASR devices, bringing new insights to both LTH theory and lightweight ASR research. Our main contributions are outlined below:
• We for the first time reveal the existence of winning tickets in the context of ASR by answering three research questions regarding LTH theory. The most lightweight winning tickets from CTC, RNN-Transducer, and Conformer backbones only possess 21.0%, 10.7%, and 8.6% remaining nonzero weights, respectively. We also show that winning tickets significantly outperform other state-of-the-art network pruning and knowledge distillation methods.
• We are the first to explore the use of structured sparsity (i.e., block sparsity (Narang et al., 2017b)) in LTH, and successfully found highly sparse winning tickets (e.g., 4.4% remaining weights) that have no performance degradation compared to using unstructured sparsity.
• Winning tickets have exceptional transferability across different datasets, which are notably better than full models. Also, the winning tickets identified from large source datasets transfer better.
• In the presence of various levels of background noise, the winning tickets consistently achieve significantly better WERs than full models. That indicates stronger noise robustness might be an extra bonus of inducing sparsity, and a missing gem by previous LTH works.

heading: 2 RELATED WORKS

text:
End-to-End Automatic Speech Recognition. Previous mainstream ASR systems are mostly based on hidden Markov model (HMM)-Gaussian mixture model (GMM) or HMM-Deep neural network
(DNN). These systems can achieve descent performance. However, such system is usually composed of modules (e.g., acoustic model, language model, lexicon) that are needed to be trained separately, which makes it hard to be optimize globally in products (Wang et al., 2019).
End-to-end ASR can directly transcribe an input audio sequence to a token (e.g., grapheme or phoneme) sequence. Current end-to-end ASR frameworks can be generally categorized into three types: CTC-based (Hannun et al., 2014; Amodei et al., 2016; Graves & Jaitly, 2014; Miao et al., 2015; Eyben et al., 2009), RNN-transducer (Graves, 2012; Graves et al., 2013; Rao et al., 2017; Dong et al., 2018b), sequence-to-sequence (Seq2Seq) model (Chorowski et al., 2015; Bahdanau et al., 2016; Chan et al., 2016; Zhang et al., 2017; Chiu et al., 2018; Prabhavalkar et al., 2018), and Transformer model (Gulati et al., 2020; Dong et al., 2018a; Wang et al., 2020b; Baevski et al., 2020).
CTC (Graves et al., 2006) is essentially a loss function, which maximizes the probability of all the paths that correspond to the ground-truth token sequence, with an augmented blank token “-” indicating no output. It avoids the need of segmental alignment/labels in training utterances, which makes tons of speech materials usable without additional annotating effort and thus fully exploits the modeling capacity of DNNs. RNN-transducer is composed of an encoder for the input audio sequence, a prediction network to model the interdependencies in between the output token sequence, and a joint network to align the input and output sequence and produce the prediction. Seq2Seq model usually has an encoder-decoder structure, where the encoder maps the input audio sequence to a hidden representation sequence and the decoder autogressively decodes the output token sequence. An attention mechanism (Chorowski et al., 2015) is trained to learn the alignment between the input and output sequences. Transformer models are similar to Seq2seq models but use multi-head attention (MHA) (Vaswani et al., 2017) layers for encoders and decoders, which has been proven to achieve the state-of-the-art ASR performance. Lottery Tickets Hypothesis. The recently emerged LTH (Frankle & Carbin, 2019b) deviates from the common wisdom of after-training pruning (Han et al., 2015), and demonstrates the existence of highly sparse subnetworks that are independently trainable from scratch, called winning tickets. Once trained, they are capable of matching or even surpassing the performance of their full models. Follow-up efforts (Renda et al., 2020; Frankle et al., 2020b) introduce the effective weight rewinding techniques to scale up LTH to large networks on large-scale datasets. LTH draws wide attention from various deep learning fields, and has been studied in image classification (Liu et al., 2019b; Evci et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al., 2020a; You et al., 2020; Ma et al., 2021; Chen et al., 2020a), natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna et al., 2020; Chen et al., 2020b;c), object detection (Girish et al., 2020), generative adversarial networks (Chen et al., 2021d; Kalibhat et al., 2020; Chen et al., 2021a), graph neural networks (Chen et al., 2021b), reinforcement learning (Yu et al., 2020), and life-long learning (Chen et al., 2021c). Several pioneer works (Mehta, 2019; Morcos et al., 2019; Desai et al., 2019; Chen et al., 2020b;a) also investigate LTH transferability across datasets and downstream tasks.
Yet to the best of our knowledge, LTH in speech models remains untouched – and that would not be a trivial extension for three aspects of reasons. On the task level, ASR has unique demands for individual user transferrability and noise robustness, which has been explained previously. On the model level, compared to CV models, speech models are mostly based on RNN backbones (Hannun et al., 2014; Chan et al., 2016), which contain recursive computational graphs and are notoriously unstable to train. That makes the pruning of RNN-based models challenging too (Zhang & Stadie, 2019), and off-the-shelf methods developed for pruning CNNs are often found ineffective or even inferior to random pruning, when applied to RNNs. Pruning methods customized for RNNs typically call for special sparse structures or stability regularizations (Narang et al., 2017a;b; Kliegl et al., 2017; Wen et al., 2018; Zhang & Stadie, 2019), and it is hence unclear whether IMP would remain stable and effective for RNN-based models in ASR. On the data level, compared to NLP models, the sequence lengths of speech signals are usually significantly larger than word embeddings (e.g., the spectrogram sequence length of a 10-second speech utterance extracted with 10ms shift is 1,000; the number of words in a sentence is usually between 10 and 50), which also inevitably costs higher computational complexity. Therefore, finding sparse subnetworks that can maintain the full model performance for ASR models is practically meaningful yet highly non-trivial.

heading: 3 PRELIMINARIES AND SETUPS

text:
Backbone Network. We investigate three ASR backbone networks that are widely used in both academia and productions: 1) CNN+LSTM (Amodei et al., 2016) model with CTC loss (Graves
et al., 2006); 2) RNN-Transducer (Graves, 2012); and 3) Convolution-augmented Transformer (Conformer) (Gulati et al., 2020). Please see Appendix A.1 for details of the backbones.
Datasets, Training, and Evaluations. We conducted experiments on three commonly used ASR corpora: TED-LIUM (Rousseau et al., 2012) (118 hours), Common Voice (Ardila et al., 2020) (582 hours), and LibriSpeech (Panayotov et al., 2015) (960 hours). Note that LibriSpeech has two test sets: test-clean – little noise inutterances; test-other – considerable noise in utterances. We test the LibriSpeech model on the two test sets hereinafter. During training, we set the batch size to 32 and an initial learning rate to 0.0003, which is annealed down by a factor of 1.1 at the end of each epoch. All the models were trained for 16 epochs. To evaluate the performance, we consider two measurements:
• Word Error Rate. WER is the standard metric measuring the accuracy of ASR models. WER is computed as: WER = (S + I +D)/N , where S, I , D, and N denote the number of substitutions, insertions, deletions, and the total number of words, respectively.
• Number of Parameters. The number of parameters measures the complexity of a model. In our case, since all subnetworks were pruned from the full models, we use the percentage of Remaining Weights as an alternative measurement. We define Sparsity as Sparsity(%) = 100%−Remaining Weights(%). Subnetworks. For a dense model f(x; θ), its subnetworks can be derived as f(x;m θ) with a binary pruning mask m ∈ {0, 1}d, where is the element-wise product and d is the dimension of pruneable model parameters. We use ADt (f(x; θ)) to represent the training algorithm (e.g., Adam (Kingma & Ba, 2017) with grid searched hyperparameters) that trains a network f(x; θ) on a dataset D (e.g., LibriSpeech) for t iterations. Let θ0 be the random initialized network weights. Subnetwork Evaluation. To measure the generalization ability of obtained subnetworks, we define ED(ADt (f(x; θ))) as the evaluation function of model f returned from ADt on the dataset D. Then, we further introduce:
? Matching subnetworks. Following the definition in (Frankle et al., 2020a; Chen et al., 2020b;a), a subnetwork f(x;m θ) is matching if it satisfies the following condition that indicates matching subnetworks achieve no worse performance than its dense counterpart under the same training algorithm ADt and evaluation metric ED: ED ( ADt (f (x;m θ)) ) ≥ ED ( ADt (f (x; θ0)) ) .
? Winning ticket. f(x;m θ) is a winning ticket for ADt , if it is (i) a matching subnetwork and (ii) θ = θ0 for ADt . ? Transferable Winning ticket. A subnetwork f(x;m θ) is transferable to target datasets {Di}Ni=1 if and only if it is a winning ticket for each ADiti . The subnetwork f(x;m θ) is derived from the source dataset Ds /∈ {Di}Ni=1. Pruning Method for Subnetwork Searching. Iterative weight magnitude pruning (IMP) is the widely used algorithm in previous LTH literature (Frankle & Carbin, 2019a; Frankle et al., 2020a; Chen et al., 2020b). To identify subnetworks f(x;m θ), IMP performs following three steps: (1) training a unpruned dense network to completion on a datasetD (i.e., applyingADt ); (2) eliminating a portion of insignificant weights with the globally smallest magnitudes (Han et al., 2015; Renda et al., 2020) so that the model only has si% of weights remaining (i.e., the sparsity); (3) rewinding model weights to θ (θ = θ0, the original random initialization; or θ = θpre, the weights from a pre-trained model) and finetuning the subnetwork to converge by leveraging ADt . Note that steps (2) and (3) usually needs to be iteratively repeated for several rounds for finding highly competitive winning tickets. In all experiments, we set si% = (1− 0.8i)× 100%, where i is the number of iterations.

heading: 4 THE EXISTENCE OF WINNING TICKETS IN SPEECH RECOGNITION

text:
In this section, we explore the existence of winning tickets in the three ASR backbones. Namely, we would like to answer the following research questions from an empirical perspective:
• RQ1: Can we find winning tickets f(x;mIMP θ) for speech recognition model using IMP? How much do they improve model complexity?
• RQ2: Do winning tickets identified by IMP have less complexities or better performance, compared to random pruning/random tickets and other compression methods?
• RQ3: Instead of using randomized weights θ0 as the initialization of IMP, does it improve the performance of winning tickets if we use weights θpre from a pre-trained model?
RQ1: Does winning tickets exist in speech recognition models? To answer the questions, we conducted experiments on three backbones and three datasets. For each trial, we first run IMP to extract a binary pruning mask at each sparsity. Then, we generate one subnetwork at each sparsity by applying the corresponding mask to the model and reset the weights to the original random initialization θ0. Finally, we train each subnetwork and computes their WER on the test set to determine if they are winning tickets. All the training hyperparameters in training a subnetwork are the same as those in training the full model.
As shown in Table 1, winning tickets can be identified on all three backbones. The most lightweight winning tickets on CTC, RNN-Transducer, and Conformer have 21.0%, 10.7%, and 8.6% remaining weights, respectively. In addition, we noticed that the RNN-Transducer and Conformer subnetworks at the extreme sparsity have less percentage of remaining weights than CTC subnetworks, likely due to these models being more over-parameterized (CTC: 86.62M parameters; RNN-Transducer: 132.23M; Conformer: 118.40M). These results show that, for a fixed dataset, the winning tickets extracted from larger models are sparser.
Similarly, winning tickets can also be identified on all three datasets, as shown in Table 2. We also found that the sparsity of a winning ticket is correlated to the size of the dataset. For example, TEDLIUM has a relatively small size (118 hours) compared to CommonVoice (582 hours) and LibriSpeech (960 hours). Accordingly, the remaining weights of TED-LIUM winning ticket are significantly lower than that of CommonVoice and LibriSpeech winning tickets. A possible explanation is models become relatively more overparameterized for smaller training sets, which allows them to be more amenable for sparsification (Li et al., 2020). Similar observations can be found in (Chen et al., 2020b).
From Table 1 and 2, another interesting finding is that the subnetworks with low sparsity (remain most of the weights) always achieve preferable performance than the full model. We also provided a visualization of the outputs from the full model, the most sparse subnetwork, and the best performing subnetwork in Figure 1 (see Appendix A.4 for more examples.). Similarly, we observed larger performance improvement on smaller datasets, possibly also due to the over parameterization issue. Results indicate that LTH is also a potential research direction in improving overall ASR performance.
RQ2: Does IMP winning tickets have lower complexity or better performance than random pruning/tickets and other compression methods? As suggested in previous studies (Frankle & Carbin, 2019a; Chen et al., 2020b; 2021d; 2020a), the two key aspects for a winning ticket to achieve the desired performance are: 1) initial weight θ, and 2) mask generated from IMP mIMP . In this subsection, we test if this argument hold in ASR winning tickets. To achieve this, we compare the winning tickets against two baseline pruning approaches: random pruning and random tickets. The subnetworks identified with random pruning are initialized with θ0 but the masks are randomly generated mRAND. By contrast, the subnetworks identified with random tickets are randomly initialized θ ′
0 but the masks are generated by IMP mIMP . As a result, with these two comparisons, we can explore if the two aspects are necessary to a winning ticket.
We tested the three pruning approaches on TED-LIUM dataset. Figure 2 shows the WER curves of the three approaches. From the figure, we found that random pruning and random ticket could find
winning tickets, but the extreme remaining weights are much higher than the one identified by IMP (IMP: 4.4%, Random Pruning: 26.2%, Random Ticket: 10.7%). Random pruning can roughly retain its performance when the remaining weights are larger than 26.2%. However, the performance drops dramatically as the remaining weights continue decreasing. Random ticket achieves very similar performance to IMP when the remaining weights are larger than 10.7%, but it degrades much faster than IMP after this point. Results verified the previous statement in ASR models that both initial weight θ and mask generated from IMP mIMP are necessary for winning tickets. Additionally, the comparison between random pruning and random ticket suggests that mIMP is more important than θ for winning tickets.
Additionally, we compared our proposed approach against four state-of-the-art neural network compression baselines: 1) Standard Pruning (Han et al., 2015; Blalock et al., 2020; Shangguan et al., 2019); 2) TutorNet (Yoon et al., 2021); 3) MLKD + multi-teacher (Li et al., 2021); 4) Sequence-level KD (Takashima et al., 2018). Baseline 1 is the most commonly used and best performing network pruning approach, which iteratively prunes the lowest magnitude weights and train the network until reaching the target sparsity (without any rewinding). Baseline 2, 3, and 4 are
three recent knowledge distillation approaches. As shown in Table 3, our winning ticket at extreme sparsity (Proposedext) achieves superior WER than all the baselines, while using less parameters.
RQ3: Can IMP find better subnetworks by initializing from a pre-trained model? Training ASR model from a good initialization usually results in more satisfactory performance (Jaitly et al., 2012), which is commonly achieved by finetuning a pre-trained model in practice. Being aware of this, we would like to investigate if IMP can find better subnetworks for ASR by initializing with weights from a pre-trained model θ = θpre. To verify this, we ran IMP on CNN+LSTM backbone with TED-LIUM dataset, where the weights were initialized from either LibriSpeech pre-trained model θpre = θLibri or CommonVoice pre-trained model θpre = θCV .
WER curves of the subnetworks in this experiment are shown in Figure 3. Both initializing from θLibri and θCV significantly improves the performance of the subnetworks at any sparsity. More importantly, we found a rapid WER increase when the remaining weights are less than 4.4% for IMP initialized from θ0. In contrast, the WERs of subnetworks in θLibri and θCV systems degrades much slower when the remaining weights are extremely low, indicating that the subnetworks that are identified from pre-trained models can utilize the parameters more efficiently. A possible reason is that the flatter loss surfaces of the pre-trained models (Liu et al., 2019a) make it more traceable for compression, leading to highly quality subnetworks, as also evidenced in (Chen et al., 2020b;a).
Table 4 shows quantitative results of these systems. Initializing from θLibri achieves the best performance among all three systems. The WERs of subnetworks initialized from either θLibri or θCV at different sparsities varies marginally, which makes it hard to find the winning ticket at extreme sparsity. Therefore, we show the WER of all the system with 4.4% remaining weights instead, corresponding to the extreme sparsity of the subnetworks initialized with θ0. Although WER4.4% of θLibri and θCV
systems are equal or higher than WERs of the corresponding full models, they are still significantly lower than θ0 system. Additionally, the remaining weights of best performing subnetworks in θLibri and θCV systems are higher than θ0, however, when reducing the amount of remaining weights, the WERs only have minimal degradations.
Summary. We conducted extensive experiments to answer the three research questions about the existence of winning tickets in three ASR backbones. First, our results verified the existence of winning tickets in CTC, RNN-Transducer, and Conformer models, even at high sparsity (e.g., 4.4% remaining weights). Second, we compared IMP with random pruning/tickets and other state-of-the-art network compression methods. The results suggest that matching subnetworks extracted by IMP significantly exceed those extracted by random pruning and random tickets in all measurements, which corroborates that both binary pruning mask and weight initialization are indispensable in finding the winning tickets. Our approach also significantly outperforms other compression approaches, achieving state-of-the-art performance on ASR model compression. Lastly, we explored the use of weights from pre-trained models to initialize IMP. We found that, in this way, IMP can identify more effective winning tickets compared to random initialization, and more importantly, these winning tickets have a considerably higher parameter efficiency. These results collectively advocate the profound benefits that LTH can bring to both server-side and on-device ASR.

heading: 5 TOWARDS PRACTICAL ASR WITH WINNING TICKETS: STUDYING STRUCTURED SPARSITY, TRANSFERABLITY, AND NOISE ROBUSTNESS

text:
After we proved the existence of winning tickets in ASR models, three more properties remain to be verified: structured sparsity, transferability, and noise robustness, which are key to ASR applications.
Study of structured sparsity As we mentioned earlier, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU Kernels (Gray et al., 2017)). Therefore, it is critical to verify if we can also identify winning tickets using structured pruning. However, the exploration of structured pruning in prior LTH studies is very limited. (You et al., 2020) is only able to identify limited winning tickets with structured sparsity at low sparsity levels (70% remaining weights), which again emphasizes the difficulty in finding computation-friendly sparse patterns.
In this experiment, we applied block sparsity (Narang et al., 2017b; Shangguan et al., 2019; Wu et al., 2021) with 1×4 block to subnetwork searching, and then evaluated the subnetworks to see if they can match the full model
performance. Results in Table 5 show that using block sparsity during subnetwork searching process lead to comparable results to using unstructured pruning. Also, we visualized the weights of winning ticket models that were discovered with unstructured sparsity and block sparsity, as shown in Figure 4. We can clearly observe 1×4 block patterns from the weight matrix pruned with block sparsity. Study of Transferability In practical scenarios, the testing utterances are directly recorded from users in the wild, which may have very different distributions from the training utterances. A common way to address this issue is through speaker adaptation (Gauvain & Lee, 1994; Leggetter & Woodland, 1995; Woodland, 2001; Liao, 2013; Li et al., 2018b; Meng et al., 2019; Weninger et al., 2019), which is usually achieved by finetuning a pre-trained model on utterances whose distributions are similar to test utterances directly, or with additional loss terms (Li et al., 2018b; Meng et al., 2019; Ding et al., 2020). In the context of winning tickets, these sparse architectures have to be transferable to new datasets so that we are able to perform speaker adaptation on these models.
To examine the transferability, we conducted the following experiment: First, we identify subnetworks f(x;mIMP θ0) on a source dataset Ds at different sparsities using IMP. Then, we re-train each subnetwork on a target dataset Dt and evaluate their performance. We tested the transferability between TED-LIUM, CommonVoice, and LibriSpeech datasets. These datasets are different in terms of recording scenario, noise level, speaker coverage, training set size, etc., and therefore, their utterances have very difference distributions. For example, TED-LIUM is created using TED talks, CommonVoice is composed of volunteer’s voice recorded through website or mobile apps, and LibriSpeech is extracted from LibriVox audio books.
Figure 5 shows the performance of subnetworks transferring to the three target datasets. Results on the three datasets consistently suggest that the winning tickets are transferable across different datasets. When the remaining weights are larger than around 26.2%, the transferring tickets can generally replicate the performance of the winning tickets identified on the target datasets. However, at extremely high sparsity, the performance of transferring tickets degrades faster than the winning tickets identified on the target datasets, corresponding to the observations from previous studies (Chen et al., 2020c; Morcos et al., 2019; Chen et al., 2021d). For example, Table 9 in Appendix A.6 shows WER and remaining weight of CommonVoice and LibriSpeech winning tickets transferring to TED-LIUM dataset at extreme sparsity. Although the remaining weights of transferring winning tickets are higher than TED-LIUM winning ticket, they are still significantly lower than the full model. We also noticed is that the winning tickets identified from a larger dataset usually have better transferability, which is in accordance with (Morcos et al., 2019).
Study of Noise Robustness The training/adaptation speech utterances are mostly collected from users, which are usually recorded from uncontrolled environments with notable background noise. Even in standard ASR benchmarks such as LibriSpeech (Panayotov et al., 2015), there are significant background noise even in its “clean” subset (Zen et al., 2019). To test the noise robustness of winning tickets, we conducted an experiment on TED-LIUM dataset. Namely, we re-trained the winning tickets identified from TED-LIUM, CommonVoice, and LibriSpeech on resynthesized TED-
LIUM dataset with background noise. We used DESED dataset (Serizel et al., 2020) as the noise source, which consists of various background sounds in domestic environments such as bell, animals, running water, speech, vacuum cleaners, etc. During resynthesis, the noise audio is added to the original speech utterance with a random noise level, which is drawn from a uniform distribution from [0, Nmax]. In addition, we considered three different noise levels: Nmax = 0 (no noise),Nmax = 0.2 (low noise), Nmax = 0.5 (high noise).
Table 6 shows WERs on TED-LIUM test set of the re-trained winning tickets. First, we found the performance of the full model is dramatically susceptible to background noise. When the noise level is increase from Nmax = 0 to Nmax = 0.5, WER increases from 15.93% to 38.21%. In contrast, WERs of TED-LIUM winning ticket at different noise level are very close, indicating that the sparse subnetwork is substaintially more robust to noise. Although the winning tickets from CommonVoice and LibriSpeech cannot reach the WER of TED-LIUM winning tickets on noise data, their WERs are
still significantly lower than full model. More essentially, we can adapt transfer tickets to any target users without the need of re-finding winning tickets, which is more feasible for practical scenarios. Our findings coincide with the ones in (Ye et al., 2019; Gui et al., 2019) that an appropriate sparsity connectivity severs as implicit regularization for network training, which produces ameliorated generalization ability on shifted data distributions such as noisy or perturbed data.
Summary In this section, we examined the structured sparsity, noise robustness, and trasferability of the winning tickets. Throughout the studies, we found that winning tickets can generalize to structured sparsity with no performance degradation. In addition, winning tickets identified from source datasets can achieve matching performance on target datasets, which verifies the transferability. Lastly, winning tickets (identified from either target dataset or source datasets) are significantly more robust to noise compared to full models, especially when the noise level is high. These results jointly demonstrate the eligibility and benefits of winning tickets in on-device ASR applications.

heading: 6 CONCLUSIONS AND FUTURE WORKS.

text: In this work, we examine and leverage lottery ticket hypothesis in speech recognition for the first time. Our extensive results show that the winning tickets are not only ultra-lightweight, but also highly transferable and (even more) noise-robust, compared to the full models. These results collectively propound the use of LTH into ASR models, bringing new insights to both LTH theory and portable ASR research. The future works to this study is to evaluate the actual hardware benefits of our found winning tickets. For example, Block-Sparse GPU Kernels (Gray et al., 2017) can achieve over 5x speed up with over 80% block sparse network, and XNNPACK (Elsen et al., 2020) can speed up networks at 70%-90% unstructured sparsity on smartphone processors, which applies to our found winning tickets. In addition, we also would like to generalize LTH compression paradigm to other speech tasks such as text-to-speech synthesis (Wang et al., 2017; Shen et al., 2018) and voice-to-voice conversion (Jia et al., 2019; Biadsy et al., 2019; Zhao et al., 2019). These new directions require further customized studies and point to new opportunities for LTH in speech research.

heading: 7 REPRODUCIBILITY STATEMENT

text:
To ensure reproducibility, we the Machine Learning Reproducibility Checklist v2.0, Apr. 7 2020 (Pineau et al., 2021).
• For all models and algorithms presented,
– A clear description of the mathematical settings, algorithm, and/or model. We clearly describe all of the settings, formulations, and algorithms in Section 3.
– A clear explanation of any assumptions. We do not make assumptions. – An analysis of the complexity (time, space, sample size) of any algorithm. We use
the number of parameters as the spatial complexity of our models, as shown in Table 1.
• For any theoretical claim,
– A clear statement of the claim. We do not make theoretical claims. – A complete proof of the claim. We do not make theoretical claims.
• For all datasets used, check if you include:
– The relevant statistics, such as number of examples. We used publicly available speech corpora in all of our experiments, and we cite the papers of them following the authors instructions, and we provide related statistics in Section 3.
– The details of train/validation/test splits. We use the default train/validation/test splits of the three public datasets, as described in Section 3.
– An explanation of any data that were excluded, and all pre-processing step. We did not exclude any data. Our pre-processing steps are explained in A.1.
– A link to a downloadable version of the dataset or simulation environment. All the datasets are publicly available, and their downloadable links can be easily found at their original papers that we cite in Section 3.
– For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control. We do not collect or release new datasets.
• For all shared code related to this work, check if you include:
– All of our code will be released after the paper was peer-reviewed, as we mentioned in Abstract. We also include the key frameworks and libraries that we used during implementation in Appendix A.2.
• For all reported experimental results, check if you include:
– The range of hyper-parameters considered, method to select the best hyperparameter configuration, and specification of all hyper-parameters used to generate results. We provide all details of the hyper-parameter we used in in Section 3 and Appendix A.1. We did not perform any hyper-parameter tuning in this work since the datasets we used are with large scale, and performing hyper-parameter tuning on large scale datasets are extremely time- and resource-consuming.
– The exact number of training and evaluation runs. One run is conducted for each experiments.
– A clear definition of the specific measure or statistics used to report results. We use the word error rate (WER) as the performance metric and the remaining weights (RW) as the complexity metric on test sets,
– A description of results with central tendency (e.g. mean) & variation (e.g. error bars). We do not report error bars.
– The average runtime for each result, or estimated energy cost. We do not report the running time or energy cost.
– A description of the computing infrastructure used. All detailed descriptions are presented in Section 3 and Appendix A.1.

heading: A APPENDIX

heading: A.1 DETAILS IN BACKBONE ARCHITECTURES

text:
Data pre-processing. The input to all the backbone networks is a sequence of 161-dimensional magnitude spectrogram (160 magnitude points plus 1 DC component), extracted with 20ms window, 10ms shift, and 320-point fast fourier transform. The ground-truth labels are represented by 28 graphemes, including 26 English characters, space, and apostrophe symbol (an extra blank token “-” is also used for CTC backbone).
CTC. The network is composed of two convolutional layers and five recurrent layers, followed by a final fully-connected layer. The first convolutional layer has 32 41×11 (in frequency×time) kernels, with 2×2 stride. The second convolutional layer has 32 21×11 kernels, with 2×1 stride. For each convlutional layers, tanh is used as nonlinearity. Following these, there are five bi-directional RNN layers, each of them having 1,024 units. A final fully-connected layer with softmax operator outputs a probability distribution over characters. The total number of parameters in the full model is 86.62M.
RNN-Transducer. In RNN-Transducer, we set the encoder and decoder to have 5 and 1 bidirectionalLSTM layers, each of them having 1,024 units. The total number of parameters in the full model is 132.23M.
Conformer. In Conformer, we used the same hyper-parameters as the Conformer(L) in [4]. The model has 17 encoder layers, each of them is composed of 8-head convolution augmented attention. The convolution kernel has a shape of 1×32, and the fully-connected layers in attention is 512- dimensional. The decoder layer is the same as RNN-Transducer, with 1,024 units. The total number of parameters in the full model is 118.40M.

heading: A.2 MORE IMPLEMENTATIONAL DETAILS

text:
As we mentioned in abstract, we will open-source all of the code once the paper is peer-reveiwed. Here, we provide key references and tools that we used during our implementation.
We implement neural network training using PyTorch library (Paszke et al., 2019). Our implementation of CTC backbone is based on:
• https://github.com/SeanNaren/deepspeech.pytorch
Our implementation of RNN-Transduce and Conformer is based on:
• https://github.com/openspeech-team/openspeech
We used PyTorch pruning library for unstructured sparsity:
• https://pytorch.org/docs/stable/generated/torch.nn.utils. prune.global_unstructured.html
We implemented pruning functions for structured sparsity based on:
• https://pytorch.org/docs/stable/generated/torch.nn.utils. prune.RandomStructured.html
• https://github.com/openai/blocksparse
• https://www.tensorflow.org/model_optimization/guide/pruning

heading: A.3 EVALUATING WINNING TICKETS OF THREE ASR BACKBONES ON LIBRISPEECH TEST-OTHER SUBSET

text: The performance of the three backbones on LibriSpeech test-other subset is shown in Table 7.

heading: A.4 MORE EXAMPLES OF THE MODEL OUTPUTS

text: We provided a couple more examples of the outputs from full model, winning tickets at extreme sparsity, and best performing winning tickets on TED-LIUM test utterances in Figure 6. The examples in the first row show the cases where the winning tickets avoid the errors appeared in the full model outputs. Meanwhile, we also noticed a few cases where winning tickets produce erroneous recognition results, as shown in the second and the third rows. Most of these utterances have either a very fast speaking rate or unclear pronunciations of words. Therefore, neither full model nor winning tickets reasonably transcribe the utterances in challenging cases, showing the limited capability of the models.

heading: A.5 COMPARISON TO OTHER MODEL COMPRESSION APPROACHES ON LIBRISPEECH TEST-OTHER SUBSET

text: Comparison between the proposed approach and state-of-the-art distillation and pruning methods on LibriSpeech test-other subset is shown in Table 8. Sequence-level KD approach (Takashima et al., 2018) does not report their evaluation results on test-other subset, and therefore, we use “N/A” in the table.

heading: A.6 MORE RESULTS IN TRANSFERRING STUDY

text: We have shown the WER curves of the transferring winning tickets transferred in Figure 5. Here, we included the WER and RW at the extreme sparsity in Table 9, Table10 and Table 11, respectively. The transferring tickets to both TED-LIUM and CommonVoice have a extreme remaining weights of 32.8%, and those to LibriSpeech have a extreme remaining weights of 32.8% compared to the full models. Although they are not as sparse as the winning tickets identified from the target datasets, the transferring tickets are still much more efficient than full models. These results are in consensus with our findings in Section 5.

heading: A.7.2 RUN TIME EVALUATIONS

heading: A.7.1 RESULTS ON THE EXISTENCE OF THE THREE BACKBONES ON TED-LIUM AND COMMONVOICE

heading: A.7 EXTRA NEW RESULTS

heading: A.7.3 UPDATED NOISE ROBUSTNESS RESULTS

heading: A.7.4 BLOCK SPARSITY EXPERIMENTS WITH LARGER BLOCK SIZE

heading: A.7.5 PSEUDO-CODE OF LTH PRUNING ALGORITHM

text: Algorithm 1 Lottery Ticket Hypothesis Pruning 1: Set the initial mask m, with the weight initialization θ. 2: repeat 3: Rewind the weight to θ 4: Train f(x;m θ) for t epochs with algorithm ADt , i.e., ADt (f(x;m θ)) 5: Prune 20% of remaining weights in ADt (f(x;m θ)) and update m accordingly 6: until the sparsity of m reaches the desired sparsity level s 7: Return f(x;m θ).

heading: A.7.6 FURTHER EXPLANATION OF BLOCK SPARSITY.

text: In terms of block sparsity, it prunes a block of parameters (e.g., 1x4 block) instead of just one parameter, which has the minimal magnitude. Unstructured pruning could be thought as block sparsity with 1x1 block.

heading: A.7.7 EXPLANATION OF THE CHOICE OF SPARSITY LEVELS

text:
In our current setting, we prune out the 20% of remaining weights that have the lowest magnitude. The reason for choosing the value to be 20% is based on considerations of model performance and computation resources. When the amount of weights pruned at each iteration becomes larger, less iterations will be needed. However, it is more likely to have larger performance regressions, since the pruned weights could be useful in the subnetwork at higher sparsity. On the other hand, if the amount of weight pruned at each iteration is too small, the number of iterations required would be very large,
which is extremely resource demanding. As a result, we feel 20% is a good one considering both factors.

title: AUDIO LOTTERY: SPEECH RECOGNITION MADE