summaries: AdaSpeech is a paper on practical TTS custom voice adaptation with the aim of reducing the amount of adapted parameters per voice to allow cloud serving of a large number of custom voices while maintaining high adaptation quality and similarity. The novel piece that enables this is the conditioning of layernorm in the model on the speaker embedding. The grammar reads slightly awkwardly in places, but the paper is understandable and well structured. Descriptions of the model, experiments, and analysis of results are well done.

summaries: The authors propose an interesting text-to-speech adaptation method for high quality and efficient customization of new voice. The proposed method consists of two-stage modeling : multi-phonetic-level acoustic condition modeling and conditional layer normalization. In the first stage modeling, the authors proposed a new phoneme-level acoustic condition modeling in addition to the speaker and utterance-level approaches. In the second stage modeling, they employ conditional layer normalization for efficient adaptation.

summaries: In this paper, the authors present AdaSpeech, a TTS system that can adapt to a custom voice with a high quality output and a low number of additional parameters. The model is based on the TTS model in FastSpeech 2, with several additional components. The authors show that AdaSpeech has improved results over other baselines. They also provide an interesting ablation study.

abstractText: Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech from her/him. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions which could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we model the acoustic information in both utterance and phoneme level. Specifically, we use one acoustic encoder to extract an utterance-level vector and another one to extract a sequence of phoneme-level vectors from the target speech during pre-training and fine-tuning; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phonemelevel vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. The audio samples are available at https://speechresearch.github.io/adaspeech/.

authors:
Sercan Arik
Jitong Chen
Kainan Peng
Wei Ping
Yanqi Zhou

title: Neural voice cloning with a few samples

venue: In Advances in Neural Information Processing Systems,

authors:
Sercan O Arik
Mike Chrzanowski
Adam Coates
Gregory Diamos
Andrew Gibiansky
Yongguo Kang
Xian Li
John Miller
Andrew Ng
Jonathan Raiman

title: Deep voice: Real-time neural text-to-speech

venue: arXiv preprint arXiv:1702.07825,

authors:
Mingjian Chen
Xu Tan
Yi Ren
Jin Xu
Hao Sun
Sheng Zhao
Tao Qin

title: Multispeech: Multispeaker text to speech with transformer

venue: Proc. Interspeech

authors:
Yutian Chen
Yannis Assael
Brendan Shillingford
David Budden
Scott Reed
Heiga Zen
Quan Wang
Luis C Cobo
Andrew Trask
Ben Laurie

title: Sample efficient adaptive text-to-speech

venue: arXiv preprint arXiv:1809.10460,

authors:
Erica Cooper
Cheng-I Lai
Yusuke Yasuda
Fuming Fang
Xin Wang
Nanxin Chen
Junichi Yamagishi

title: Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings

authors:
Andrew Gibiansky
Sercan Arik
Gregory Diamos
John Miller
Kainan Peng
Wei Ping
Jonathan Raiman
Yanqi Zhou

title: Deep voice 2: Multi-speaker neural text-to-speech

venue: In Advances in neural information processing systems,

title: The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/, 2017

authors:
Ye Jia
Yu Zhang
Ron Weiss
Quan Wang
Jonathan Shen
Fei Ren
Patrick Nguyen
Ruoming Pang
Ignacio Lopez Moreno
Yonghui Wu

title: Transfer learning from speaker verification to multispeaker text-to-speech synthesis

authors:
Jaehyeon Kim
Sungwon Kim
Jungil Kong
Sungroh Yoon

title: Glow-tts: A generative flow for text-to-speech via monotonic alignment search

venue: arXiv preprint arXiv:2005.11129,

authors:
Zvi Kons
Slava Shechtman
Alex Sorin
Carmel Rabinovitz
Ron Hoory

title: High quality, lightweight and adaptable tts using lpcnet

authors:
Kundan Kumar
Rithesh Kumar
Thibault de Boissiere
Lucas Gestin
Wei Zhen Teoh
Jose Sotelo
Alexandre de Brébisson
Yoshua Bengio
Aaron C Courville

title: Melgan: Generative adversarial networks for conditional waveform synthesis

authors:
Chao Li
Xiaokong Ma
Bing Jiang
Xiangang Li
Xuewei Zhang
Xiao Liu
Ying Cao
Ajay Kannan
Zhenyao Zhu

title: Deep speaker: an end-to-end neural speaker embedding system

venue: arXiv preprint arXiv:1705.02304,

authors:
Renqian Luo
Xu Tan
Rui Wang
Tao Qin
Jinzhu Li
Sheng Zhao
Enhong Chen
Tie-Yan Liu

title: Lightspeech: Lightweight and fast text to speech with neural architecture search

authors:
Laurens van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

authors:
Michael McAuliffe
Michaela Socolof
Sarah Mihuc
Michael Wagner
Morgan Sonderegger

title: Montreal forced aligner: Trainable text-speech alignment using kaldi

venue: In Interspeech,

authors:
Henry B Moss
Vatsal Aggarwal
Nishant Prateek
Javier González
Roberto Barra-Chicote

title: Boffin tts: Few-shot speaker adaptation by bayesian optimization

authors:
Vassil Panayotov
Guoguo Chen
Daniel Povey
Sanjeev Khudanpur

title: Librispeech: an asr corpus based on public domain audio books

venue: In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Kainan Peng
Wei Ping
Zhao Song
Kexin Zhao

title: Non-autoregressive neural text-to-speech

authors:
Wei Ping
Kainan Peng
Andrew Gibiansky
Sercan O. Arik
Ajay Kannan
Sharan Narang
Jonathan Raiman
John Miller

title: Deep voice 3: 2000-speaker neural text-to-speech

venue: In International Conference on Learning Representations,

authors:
Yi Ren
Yangjun Ruan
Xu Tan
Tao Qin
Sheng Zhao
Zhou Zhao
Tie-Yan Liu

title: Fastspeech: Fast, robust and controllable text to speech

venue: NeurIPS,

authors:
Yi Ren
Chenxu Hu
Xu Tan
Tao Qin
Sheng Zhao
Zhou Zhao
Tie-Yan Liu

title: Fastspeech 2: Fast and high-quality end-to-end text-to-speech

venue: In ICLR,

authors:
Jonathan Shen
Ruoming Pang
Ron J Weiss
Mike Schuster
Navdeep Jaitly
Zongheng Yang
Zhifeng Chen
Yu Zhang
Yuxuan Wang
Rj Skerrv-Ryan

title: Natural tts synthesis by conditioning wavenet on mel spectrogram predictions

authors:
Guangzhi Sun
Yu Zhang
Ron J Weiss
Yuan Cao
Heiga Zen
Andrew Rosenberg
Bhuvana Ramabhadran
Yonghui Wu

title: Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior

authors:
Hao Sun
Xu Tan
Jun-Wei Gan
Hongzhi Liu
Sheng Zhao
Tao Qin
Tie-Yan Liu

title: Token-level ensemble distillation for grapheme-to-phoneme conversion

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Christophe Veaux
Junichi Yamagishi
Kirsten MacDonald

title: Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning

authors:
Li Wan
Quan Wang
Alan Papir
Ignacio Lopez Moreno

title: Generalized end-to-end loss for speaker verification

authors:
Yuxuan Wang
RJ Skerry-Ryan
Daisy Stanton
Yonghui Wu
Ron J Weiss
Navdeep Jaitly
Zongheng Yang
Ying Xiao
Zhifeng Chen
Samy Bengio

title: Tacotron: Towards end-to-end speech synthesis

venue: arXiv preprint arXiv:1703.10135,

authors:
Yuzi Yan
Xu Tan
Bohan Li
Tao Qin
Sheng Zhao
Yuan Shen
Tie-Yan Liu

title: Adaspeech 2: Adaptive text to speech with untranscribed data

authors:
Heiga Zen
Viet Dang
Rob Clark
Yu Zhang
Ron J Weiss
Ye Jia
Zhifeng Chen
Yonghui Wu

title: Libritts: A corpus derived from librispeech for text-to-speech

authors:
Zhen Zeng
Jianzong Wang
Ning Cheng
Jing Xiao

title: Prosody learning mechanism for speech synthesis system without text length limit

venue: arXiv preprint arXiv:2008.05656,

authors:
Chen Zhang
Yi Ren
Xu Tan
Jinglin Liu
Kejun Zhang
Tao Qin
Sheng Zhao
Tie-Yan Liu

title: Denoispeech: Denoising text to speech with frame-level noise modeling

authors:
Zewang Zhang
Qiao Tian
Heng Lu
Ling-Hui Chen
Shan Liu

title: Adadurian: Few-shot adaptation for neural text-to-speech with durian

venue: arXiv preprint arXiv:2005.05642,

heading: 1 INTRODUCTION

text:
Text to speech (TTS) aims to synthesize natural and intelligible voice from text, and attracts a lot of interests in machine learning community (Arik et al., 2017; Wang et al., 2017; Gibiansky et al., 2017; Ping et al., 2018; Shen et al., 2018; Ren et al., 2019). TTS models can synthesize natural human voice when training with a large amount of high-quality and single-speaker recordings (Ito, 2017), and has been extended to multi-speaker scenarios (Gibiansky et al., 2017; Ping et al., 2018; Zen et al., 2019; Chen et al., 2020) using multi-speaker corpora (Panayotov et al., 2015; Veaux et al., 2016; Zen et al., 2019). However, these corpora contain a fixed set of speakers where each speaker still has a certain amount of speech data.
Nowadays, custom voice has attracted increasing interests in different application scenarios such as personal assistant, news broadcast and audio navigation, and has been widely supported in commercial speech platforms (some custom voice services include Microsoft Azure, Amazon AWS and Google Cloud). In custom voice, a source TTS model is usually adapted on personalized voices with few adaptation data, since the users of custom voice prefer to record as few adaptation data as possible (several minutes or seconds) for convenient purpose. Few adaptation data presents great challenges
∗The first two authors contribute equally to this work. Corresponding author: Xu Tan, xuta@microsoft.com.
on the naturalness and similarity of adapted voice. Furthermore, there are also several distinctive challenges in custom voice: 1) The recordings of the custom users are usually of different acoustic conditions from the source speech data (the data to train the source TTS model). For example, the adaptation data is usually recorded with diverse speaking prosodies, styles, emotions, accents and recording environments. The mismatch in these acoustic conditions makes the source model difficult to generalize and leads to poor adaptation quality. 2) When adapting the source TTS model to a new voice, there is a trade-off between the fine-tuning parameters and voice quality. Generally speaking, more adaptation parameters will usually result in better voice quality, which, as a result, increases the memory storage and serving cost1.
While previous works in TTS adaptation have well considered the few adaptation data setting in custom voice, they have not fully addressed the above challenges. They fine-tune the whole model (Chen et al., 2018; Kons et al., 2019) or decoder part (Moss et al., 2020; Zhang et al., 2020), achieving good quality but causing too many adaptation parameters. Reducing the amount of adaptation parameters is necessary for the deployment of commercialized custom voice. Otherwise, the memory storage would explode as the increase of users. Some works only fine-tune the speaker embedding (Arik et al., 2018; Chen et al., 2018), or train a speaker encoder module (Arik et al., 2018; Jia et al., 2018; Cooper et al., 2020; Li et al., 2017; Wan et al., 2018) that does not need fine-tuning during adaptation. While these approaches lead a light-weight and efficient adaptation, they result in poor adaptation quality. Moreover, most previous works assume the source speech data and adaptation data are in the same domain and do not consider the setting with different acoustic conditions, which is not practical in custom voice scenarios.
In this paper, we propose AdaSpeech, an adaptive TTS model for high-quality and efficient customization of new voice. AdaSpeech employ a three-stage pipeline for custom voice: 1) pre-training; 2) fine-tuning; 3) inference. During the pre-training stage, the TTS model is trained on large-scale multi-speaker datasets, which can ensure the TTS model to cover diverse text and speaking voices that is helpful for adaptation. During the fine-tuning stage, the source TTS model is adapted on a new voice by fine-tuning (a part of) the model parameters on the limited adaptation data with diverse acoustic conditions. During the inference stage, both the unadapted part (parameters shared by all custom voices) and the adapted part (each custom voice has specific adapted parameters) of the TTS model are used for the inference request. We build AdaSpeech based on the popular non-autoregressive TTS models (Ren et al., 2019; Peng et al., 2020; Kim et al., 2020; Ren et al., 2021) and further design several techniques to address the challenges in custom voice:
• Acoustic condition modeling. In order to handle different acoustic conditions for adaptation, we model the acoustic conditions in both utterance and phoneme level in pre-training and fine-tuning. Specifically, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech, which are taken as the input of the mel-spectrogram decoder to represent the global and local acoustic conditions respectively. In this way, the decoder can predict speech in different acoustic conditions based on these acoustic information. Otherwise, the model would memorize the acoustic conditions and cannot generalize well. In inference, we extract the utterance-level vector from a reference speech and use another acoustic predictor that is built upon the phoneme encoder to predict the phoneme-level vectors.
• Conditional layer normalization. To fine-tune as small amount of parameters as possible while ensuring the adaptation quality, we modify the layer normalization (Ba et al., 2016) in the melspectrogram decoder in pre-training, by using speaker embedding as the conditional information to generate the scale and bias vector in layer normalization. In fine-tuning, we only adapt the parameters related to the conditional layer normalization. In this way, we can greatly reduce adaptation parameters and thus memory storage2 compared with fine-tuning the whole model, but maintain high-quality adaptation voice thanks to the flexibility of conditional layer normalization.
To evaluate the effectiveness of our proposed AdaSpeech for custom voice, we conduct experiments to train the TTS model on LibriTTS datasets and adapt the model on VCTK and LJSpeech datasets with different adaptation settings. Experiment results show that AdaSpeech achieves better adaptation quality in terms of MOS (mean opinion score) and SMOS (similarity MOS) than baseline methods, with
1For example, to support one million users in a cloud speech service, if each custom voice consumes 100MB model sizes, the total memory storage would be about 100PB, which is quite a big serving cost.
2We further reduce the memory usage in inference as described in Section 2.3.
only about 5K specific parameters for each speaker, demonstrating its effectiveness for custom voice. Audio samples are available at https://speechresearch.github.io/adaspeech/.
2 ADASPEECH
In this section, we first describe the overall design of our proposed AdaSpeech, and then introduce the key techniques to address the challenges in custom voice. At last, we list the pre-training, finetuning and inference pipeline of AdaSpeech for custom voice.
The model structure of AdaSpeech is shown in Figure 1. We adopt FastSpeech 2 (Ren et al., 2021) as the model backbone considering the FastSpeech (Ren et al., 2019; 2021) series are one of the most popular models in non-autoregressive TTS. The basic model backbone consists of a phoneme encoder, a mel-spectrogram decoder, and a variance adaptor which provides variance information including duration, pitch and energy into the phoneme hidden sequence following Ren et al. (2021). As shown in Figure 1, we design two additional components to address the distinctive challenges in custom voice: 1) to support diverse customers, we use acoustic condition modeling to capture the diverse acoustic conditions of adaptation speech in different granularities; 2) to support a large number of customers with affordable memory storage, we use conditional layer
normalization in decoder for efficient adaptation with few parameters while high voice quality. In the next subsections, we introduce the details of these components respectively.

heading: 2.1 ACOUSTIC CONDITION MODELING

text:
In custom voice, the adaptation data can be spoken with diverse prosodies, styles, accents, and can be recorded under various environments, which can make the acoustic conditions far different from that in source speech data. This presents great challenges to adapt the source TTS model, since the source speech cannot cover all the acoustic conditions in custom voice. A practical way to alleviate this issue is to improve the adaptability (generalizability) of source TTS model. In text to speech, since the input text lacks enough acoustic conditions (such as speaker timbre, prosody and recording environments) to predict the target speech, the model tends to memorize and overfit on the training data (Ren et al., 2021), and has poor generalization during adaptation. A natural way to solve such problem is to provide corresponding acoustic conditions as input to make the model learn reasonable text-to-speech mapping towards better generalization instead of memorizing.
To better model the acoustic conditions with different granularities, we categorize the acoustic conditions in different levels as shown in Figure 2a: 1) speaker level, the coarse-grained acoustic conditions to capture the overall characteristics of a speaker; 2) utterance level, the fine-grained acoustic conditions in each utterance of a speaker; 3) phoneme level, the more fine-grained acoustic conditions in each phoneme of an utterance, such as accents on specific phonemes, pitches, prosodies and temporal environment noises3. Since speaker ID (embedding) is widely used to capture speakerlevel acoustic conditions in multi-speaker scenario (Chen et al., 2020), speaker embedding is used by default. We describe the utterance-level and phoneme-level acoustic condition modeling as follows.
• Utterance Level. We use an acoustic encoder to extract a vector from a reference speech, similar to Arik et al. (2018); Jia et al. (2018); Cooper et al. (2020), and then expand and add it to the phoneme hidden sequence to provide the utterance-level acoustic conditions. As shown in Figure 2b, the acoustic encoder consists of several convolutional layers and a mean pooling layer to get a single vector. The reference speech is the target speech during training, while a randomly chosen speech of this speaker during inference.
• Phoneme Level. We use another acoustic encoder (shown in Figure 2c) to extract a sequence of phoneme-level vectors from the target speech and add it to the phoneme hidden sequence to
3Generally, more fine-grained frame-level acoustic conditions (Zhang et al., 2021) exist, but have marginal benefits considering their prediction difficulty. Similarly, more coarse-grained language level conditions also exist, but we do not consider multilingual setting in this work and leave it for future work.
provide the phoneme-level acoustic conditions4. In order to extract phoneme-level information from speech, we first average the speech frames corresponding to the same phoneme according to alignment between phoneme and mel-spectrogram sequence (shown in Figure 2a), to convert to length of speech frame sequence into the length of phoneme sequence, similar to Sun et al. (2020); Zeng et al. (2020). During inference, we use another phoneme-level acoustic predictor (shown in Figure 2d) which is built upon the original phoneme encoder to predict the phoneme-level vectors.
Using speech encoders to extract a single vector or a sequence of vectors to represent the characteristics of a speech sequence has been adopted in previous works (Arik et al., 2018; Jia et al., 2018; Cooper et al., 2020; Sun et al., 2020; Zeng et al., 2020). They usually leverage them to improve the speaker timbre or prosody of the TTS model, or improve the controllability of the model. The key contribution in our acoustic condition modeling in this work is the novel perspective to model the diverse acoustic conditions in different granularities to make the source model more adaptable to different adaptation data. As analyzed in Section 4.2, utterance-level and phoneme-level acoustic modeling can indeed help the learning of acoustic conditions and is critical to ensure the adaptation quality.
2.2 CONDITIONAL LAYER NORMALIZATION 𝑠𝑐𝑎𝑙𝑒 ∗ 𝑋 −𝑚𝑒𝑎𝑛
𝑣𝑎𝑟 + 𝑏𝑖𝑎𝑠
Speaker Embedding
Linear Layer
Speaker ID
Linear Layer
Figure 3: Conditional LayerNorm.
Achieving high adaptation quality while using small adaptation parameters is challenging. Previous works use zero-shot adaptation with speaker encoder (Arik et al., 2018; Jia et al., 2018; Cooper et al., 2020) or only fine-tune the speaker embedding cannot achieve satisfied quality. Can we greatly increase the voice quality at the cost of slightly more but negligible parameters? To this end, we analyze the model parameters of FastSpeech 2 (Ren et al., 2021), which is basically built upon
the structure of Transformer (Vaswani et al., 2017), with a self-attention network and a feed-forward network in each Transformer block. Both the matrix multiplications in the query, key, value and output of self-attention and two-layer feed-forward networks are parameter-intensive, which is not efficient to adapt. We find that layer normalization (Ba et al., 2016) is adopted in each self-attention and feed-forward network in decoder, which can greatly influence the hidden activation and final prediction with a light-weight learnable scale vector γ and bias vector β: LN(x) = γ x−µσ +β, where µ and σ are the mean and variance of hidden vector x.
4Note that although the extracted vectors can contain all phoneme-level acoustic conditions ideally, we still use pitch and energy in the variance adaptor (shown in Figure 1) as additional input following Ren et al. (2021), in order to ease the burden of acoustic condition learning and focus on learning other acoustic conditions. We also tried to remove pitch and energy but found it causes worse adaptation quality.
If we can determine the scale and bias vector in layer normalization with the corresponding speaker characteristics using a small conditional network, then we can fine-tune this conditional network when adapting to a new voice, and greatly reduce the adaptation parameters while ensuring the adaptation quality. As shown in Figure 3, the conditional network consists of two simple linear layers W γc and W βc that take speaker embedding E s as input and output the scale and bias vector respectively:
γsc = E s ∗W γc , βsc = Es ∗W βc , (1)
where s denotes the speaker ID, and c ∈ [C] denotes there are C conditional layer normalizations in the decoder (the number of decoder layer is (C − 1)/2 since each layer has two conditional layer normalizations corresponding to self-attention and feed-forward network in Transformer, and there is an additional layer normalization at the final output) and each uses different conditional matrices.

heading: 2.3 PIPELINE OF ADASPEECH

text:
We list the pre-training, fine-tuning and inference pipeline of AdaSpeech in Algorithm 1. During fine-tuning, we only fine-tune the two matrices W γc and W β c in each conditional layer normalization in decoder and the speaker embeddingEs, fixing other model parameters including the utterance-level and phoneme-level acoustic encoders and phoneme-level acoustic predictor as described in Section 2.1. During inference, we do not directly use the two matrices W γc and W β c in each conditional layer normalization since they still have large parameters. Instead we use the two matrices to calculate each scale and bias vector γsc and β s c from speaker embedding Es according to Equation 1 considering Es is fixed in inference. In this way, we can save a lot of memory storage5.
Algorithm 1 Pre-training, fine-tuning and inference of AdaSpeech 1: Pre-training: Train the AdaSpeech model θ with source training data D. 2: Fine-tuning: Fine-tuneW γc andW βc in each conditional layer normalization c ∈ [C] and speaker
embedding Es with the adaptation data Ds for each custom speaker/voice s. 3: Inference: Deployment: 1) Calculate γsc , βsc in each conditional layer normalization c ∈ [C],
and get the parameters θs = {{γsc , βsc}Cc=1, Es} for speaker s. 2) Deploy the shared model parameters θ̃ (not fine-tuned in θ during adaptation) and speaker specific parameters θs for s. Inference: Use θ̃ and θs to synthesize custom voice for speaker s.

heading: 3 EXPERIMENTAL SETUP

text:
Datasets We train the AdaSpeech source model on LibriTTS (Zen et al., 2019) dataset, which is a multi-speaker corpus (2456 speakers) derived from LibriSpeech (Panayotov et al., 2015) and contains 586 hours speech data. In order to evaluate AdaSpeech in custom voice scenario, we adapt the source model to the voices in other datasets including VCTK (Veaux et al., 2016) (a multi-speaker datasets with 108 speakers and 44 hours speech data) and LJSpeech (Ito, 2017) (a single-speaker high-quality dataset with 24 hours speech data), which have different acoustic conditions from LibriTTS. As a comparison, we also adapt the source model to the voices in the same LibriTTS dataset.
We randomly choose several speakers (including both male and female) from the training set of LibriTTS and VCTK and the only single speaker from the training set of LJSpeech for adaptation. For each chosen speaker, we randomly choose K = 20 sentences for adaptation and also study the effects of smaller K in experiment part. We use all the speakers in the training set of LibriTTS (exclude those chosen for adaptation) to train the source AdaSpeech model, and use the original test sets in these datasets corresponding to the adaptation speakers to evaluate the adaptation voice quality.
We conduct the following preprocessing on the speech and text data in these corpora: 1) convert the sampling rate of all speech data to 16kHz; 2) extract the mel-spectrogram with 12.5ms hop size and 50ms window size following the common practice in Shen et al. (2018); Ren et al. (2019); 3) convert
5Assume the dimension of speaker embedding and hidden vector are both h, the number of conditional layer normalization is C. Therefore, the number of adaptation parameters are 2h2C + h, where the first 2 represents the two matrices for scale and bias vectors, and the second term h represents the speaker embedding. If h = 256 and C = 9, the total number of parameters are about 1.2M, which is much smaller compared the whole model (31M). During deployment for each custom voice, the total additional model parameters for a new voice that need to be stored in memory becomes 2hC + h, which is extremely small (4.9K in the above example).
text sequence into phoneme sequence with grapheme-to-phoneme conversion (Sun et al., 2019) and take phoneme as the encoder input.
Model Configurations The model of AdaSpeech follows the basic structure in FastSpeech 2 (Ren et al., 2021), which consists of 4 feed-forward Transformer blocks for the phoneme encoder and melspectrogram decoder. The hidden dimension (including the phoneme embedding, speaker embedding, the hidden in self-attention, and the input and output hidden of feed-forward network) is set to 256. The number of attention heads, the feed-forward filter size and kernel size are set to 2, 1024 and 9 respectively. The output linear layer converts the 256-dimensional hidden into 80-dimensional mel-spectrogram. Other model configurations follow Ren et al. (2021) unless otherwise stated.
The phoneme-level acoustic encoder (Figure 2c) and predictor (Figure 2d) share the same structure, which consists of 2 convolutional layers with filter size and kernel size of 256 and 3 respectively, and a linear layer to compress the hidden to a dimension of 4 (we choose the dimension of 4 according to our preliminary study and is also consistent with previous works (Sun et al., 2020; Zeng et al., 2020)). We use MFA (McAuliffe et al., 2017) to extract the alignment between the phoneme and mel-spectrogram sequence, which is used to prepare the input of the phoneme-level acoustic encoder. We also tried to leverage VQ-VAE (Sun et al., 2020) into the phoneme-level acoustic encoder but found no obvious gains. The utterance-level acoustic encoder consists of 2 convolutional layers with filter size, kernel size and stride size of 256, 5 and 3, and a pooling layer to obtain a single vector.
Training, Adaptation and Inference In the source model training process, we first train AdaSpeech for 60,000 steps, and all the model parameters are optimized except the parameters of phoneme-level acoustic predictor. Then we train AdaSpeech and the phoneme-level acoustic predictor jointly for the remaining 40,000 steps, where the output hidden of the phoneme-level acoustic encoder is used as the label (the gradient is stopped to prevent flowing back to the phoneme-level acoustic encoder) to train the phoneme-level acoustic predictor with mean square error (MSE) loss. We train AdaSpeech on 4 NVIDIA P40 GPUs and each GPU has a batch size of about 12,500 speech frames. Adam optimizer is used with β1 = 0.9, β2 = 0.98, = 10−9.
In the adaptation process, we fine-tune AdaSpeech on 1 NVIDIA P40 GPU for 2000 steps, where only the parameters of speaker embedding and conditional layer-normalization are optimized. In the inference process, the utterance-level acoustic conditions are extracted from another reference speech of the speaker, and the phoneme-level acoustic conditions are predicted from phoneme-level acoustic predictor. We use MelGAN (Kumar et al., 2019) as the vocoder to synthesize waveform from the generated mel-spectrogram.

heading: 4 RESULTS

text: In this section, we first evaluate the quality of the adaptation voices of AdaSpeech, and conduct ablation study to verify the effectiveness of each component in AdaSpeech, and finally we show some analyses of our method.

heading: 4.1 THE QUALITY OF ADAPTATION VOICE

text:
We evaluate the quality of adaption voices in terms of naturalness (how the synthesized voices sound natural like human) and similarity (how the synthesized voices sound similar to this speaker). Therefore, we conduct human evaluations with MOS (mean opinion score) for naturalness and SMOS (similarity MOS) for similarity. Each sentence is listened by 20 judgers. For VCTK and LibriTTS, we average the MOS and SMOS scores of multiple adapted speakers as the final scores. We compare AdaSpeech with several settings: 1) GT, the ground-truth recordings; 2) GT mel + Vocoder, using ground-truth mel-spectrogram to synthesize waveform with MelGAN vocoder; 3) Baseline (spk emb), a baseline system based on FastSpeech2 which only fine-tunes the speaker embedding during adaptation, and can be regarded as our lower bound; 4) Baseline (decoder), another baseline system based on FastSpeech2 which fine-tunes the whole decoder during adaptation, and can be regarded as a strong comparable system since it uses more parameters during adaptation; 5) AdaSpeech, our proposed AdaSpeech system with utterance-/phoneme-level acoustic condition modeling and conditional layer normalization during adaptation6.
6The audio samples are available at https://speechresearch.github.io/adaspeech/
The MOS and SMOS results are shown in Table 1. We have several observations: 1) Adapting the model (trained on LibriTTS) to the cross-domain datasets (LJSpeech and VCTK) is more difficult than adapting to the in-domain datasets (LibriTTS), since the MOS and SMOS gap between the adaptation models (two baselines and AdaSpeech) and the ground-truth mel + vocoder setting is bigger on cross-domain datasets7. This also confirms the challenges of modeling different acoustic conditions in custom voice scenarios. 2) Compared with only fine-tuning speaker embedding, i.e., Baseline (spk emb), AdaSpeech achieves significant improvements in terms of both MOS and SMOS in the three adaptation datasets, by only leveraging slightly more parameters in conditional layer normalization. We also analyze in next subsection (Table 3) that even if we increase the adaptation parameters of baseline to match or surpass that in AdaSpeech, it still performs much worse than AdaSpeech. 3) Compared with fine-tuning the whole decoder, i.e., Baseline (decoder), AdaSpeech achieves slightly better quality in both MOS and SMOS and importantly with much smaller adaptation parameters, which demonstrates the effectiveness and efficiency of our proposed acoustic condition modeling and conditional layer normalization. Note that fine-tuning the whole decoder causes too much adaptation parameters that cannot satisfy the custom voice scenario.
4.2 METHOD ANALYSIS Setting CMOS AdaSpeech 0
AdaSpeech w/o UL-ACM −0.12 AdaSpeech w/o PL-ACM −0.21 AdaSpeech w/o CLN −0.14
Table 2: The CMOS of the ablation study on VCTK. UL-ACM and PL-ACM represents utterance-level and phoneme-level acoustic condition modeling, and CLN represents conditional layer normalization.
In this section, we first conduct ablation studies to verify the effectiveness of each component in AdaSpeech, including utterance-level and phonemelevel acoustic condition modeling, and conditional layer normalization, and then conduct more detailed analyses on our proposed AdaSpeech.
Ablation Study We compare the CMOS (comparison MOS) of the adaptation voice quality when removing each component in AdaSpeech on VCTK testset (each sentence is listened by 20 judgers). Specifically, when removing conditional layer normalization,
we only fine-tune the speaker embedding. From Table 2, we can see that removing utterance-level and phoneme-level acoustic modeling, and conditional layer normalization all result in performance drop in voice quality, demonstrating the effectiveness of each component in AdaSpeech.
Analyses on Acoustic Condition Modeling We analyze the vectors extracted from the utterancelevel acoustic encoder for several speakers on LibriTTS datasets. We use t-SNE (Maaten & Hinton,
7For example, the MOS gaps of the three settings (two baselines and AdaSpeech) on LJSpeech are 1.38, 0.31, 0.30, and on VCTK are 1.38, 0.39, 0.35, respectively, which are bigger than that on LibriTTS (0.63, 0.14, 0.10).
2008) to illustrate them in Figure 4a, where each point represents an utterance-level vector and each color belongs to the same speaker. It can be seen that different utterances of the same speaker are clustered together but have difference in acoustic conditions. There are some exceptions, such as the two pink points one blue point in the brown solid circle. According to our investigation on the corresponding speech data, these points correspond to the utterances with short and emotional voice, and thus are close to each other although belonging to different speakers.
Analyses on Conditional Layer Normalization We further compare conditional layer normalization (CLN) with other two settings: 1) LN + fine-tune scale/bias: removing the condition on speaker embedding, and only fine-tuning scale/bias in layer normalization and speaker embedding; 2) LN + fine-tuning others: removing the condition on speaker embedding, and instead fine-tuning other (similar or even larger amount of) parameters in the decoder8. The CMOS evaluations are shown in Table 3. It can be seen that both settings result in worse quality
compared with conditional layer normalization, which verifies its effectiveness.
Varying Adaptation Data We study the voice quality with different amount of adaptation data (fewer than the default setting) on VCTK and LJSpeech, and conduct MOS evaluation as shown in Figure 4b. It can be seen that the voice quality continue drops when adaptation data decreases, and drops quickly when the adaptation data is fewer than 10 sentences.

heading: 5 CONCLUSIONS

text:
In this paper, we have developed AdaSpeech, an adaptive TTS system to support the distinctive requirements in custom voice. We propose acoustic condition modeling to make the source TTS model more adaptable for custom voice with various acoustic conditions. We further design conditional layer normalization to improve the adaptation efficiency: fine-tuning few model parameters to achieve high voice quality. We finally present the pipeline of pre-training, fine-tuning and inference in AdaSpeech for custom voice. Experiment results demonstrate that AdaSpeech can support custom voice with different acoustic conditions with few memory storage and at the same time with high voice quality. For future work, we will further improve the modeling of acoustic conditions in the source TTS model and study more diverse acoustic conditions such as noisy speech in custom voice. We will also investigate the adaptation setting with untranscribed data (Yan et al., 2021) and further compress the model size (Luo et al., 2021) to support more custom voices.
8According to the preliminary study, we found fine-tuning the last linear layer and the last feed-forward network in decoder can result in better performance than fine-tuning other part in decoder.


summaries: This paper proposes effective gradient flow (EGF), which is a layer-wise normalized gradient flow. Compared to (unnormalized) gradient flow, the paper shows that the proposed EGF is (slightly) better correlated with metrics like test loss and test accuracy (see Table 1). Given that this claim is supported with experimental results, the paper would become much stronger if a larger number and a more diverse set of data-sets were used (in addition to CIFAR-10 and CIFAR-100, which are two very similar image-data-sets) as to show that the claim holds more generally. Apart from that, given that the correlations are (only) about 0.4 in Table 1, it seems that only some aspects are explained by EGF.

summaries: Recently, initialization has been found critical to the model accuracy attained by sparse training [1]. In this paper, the authors studies the impact of factors other than initialization on the model accuracy attained by sparse training relative to dense training (under the same model parameter count). At the core of this paper, the authors argue that the effective gradient flow (grad norm from only activate model weight dimensions) is an effective indicator on the model accuracy attained by sparse training. Firstly, the authors show that the effective gradient flow attains higher correlation with model accuracy than the norm of full gradient (including gradients on sparsified weight dimensions) in sparse training. Secondly, the paper empirically demonstrate that in sparse training, 1) weight decay and data augmentation can hurt model accuracy, 2) batch normalization plays significantly role for model accuracy in sparse training and 3) non-saturating activations boost the magnitude of effective gradient flow and consequently improve model accuracy.

summaries: This paper looks at how choices in optimization affect how well you can train sparse networks. The authors come up with a new measure of effective gradient flow, which is important for good performance. They also compare sparse vs. dense networks across various optimizers, hyperparameters and activation functions, and find that batch norm and certain activation functions are beneficial for sparse networks

abstractText: Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider various choices made during training that might disadvantage sparse networks. We measure the gradient flow across different networks and datasets, and show that the default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks. Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results.

authors:
Guillaume Bellec
David Kappel
Wolfgang Maass
Robert Legenstein

title: Deep rewiring: Training very sparse deep networks

venue: arXiv preprint arXiv:1711.05136,

authors:
Yoshua Bengio
Patrice Simard
Paolo Frasconi

title: Learning long-term dependencies with gradient descent is difficult

venue: IEEE transactions on neural networks,

authors:
Zhao Chen
Vijay Badrinarayanan
Chen-Yu Lee
Andrew Rabinovich

title: Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks

venue: In International Conference on Machine Learning,

authors:
Djork-Arné Clevert
Thomas Unterthiner
Sepp Hochreiter

title: Fast and accurate deep network learning by exponential linear units (elus)

venue: arXiv preprint arXiv:1511.07289,

authors:
Yann Le Cun
John S. Denker
Sara A. Solla

title: Optimal brain damage

title: Statistical comparisons of classifiers over multiple data sets

venue: Journal of Machine learning research,

authors:
Tim Dettmers
Luke Zettlemoyer

title: Sparse networks from scratch: Faster training without losing performance

venue: arXiv preprint arXiv:1907.04840,

title: Activation function impact on sparse neural networks

venue: B.S. thesis, University of Twente,

authors:
John Duchi
Elad Hazan
Yoram Singer

title: Adaptive subgradient methods for online learning and stochastic optimization

authors:
Ronen Eldan
Ohad Shamir

title: The power of depth for feedforward neural networks

venue: In Conference on learning theory,

authors:
Utku Evci
Trevor Gale
Jacob Menick
Pablo Samuel Castro
Erich Elsen

title: Rigging the Lottery: Making All Tickets Winners

venue: arXiv e-prints, art

authors:
Utku Evci
Fabian Pedregosa
Aidan Gomez
Erich Elsen

title: The difficulty of training sparse neural networks

venue: arXiv preprint arXiv:1906.10732,

authors:
Utku Evci
Yani A Ioannou
Cem Keskin
Yann Dauphin

title: Gradient flow in sparse neural networks and how lottery tickets win

venue: arXiv preprint arXiv:2010.03533,

authors:
Jonathan Frankle
Michael Carbin

title: The lottery ticket hypothesis: Finding sparse, trainable neural networks

venue: arXiv preprint arXiv:1803.03635,

authors:
Jonathan Frankle
Gintare Karolina Dziugaite
Daniel M Roy
Michael Carbin

title: Linear mode connectivity and the lottery ticket hypothesis

venue: arXiv preprint arXiv:1912.05671,

title: Stabilizing the lottery ticket hypothesis

venue: arXiv preprint arXiv:1903.01611,

authors:
Jonathan Frankle
Gintare Karolina Dziugaite
Daniel M. Roy
Michael Carbin

title: Pruning neural networks at initialization: Why are we missing the mark

title: On the approximate realization of continuous mappings by neural networks

venue: Neural networks,

authors:
Trevor Gale
Erich Elsen
Sara Hooker

title: The State of Sparsity in Deep Neural Networks

title: The state of sparsity in deep neural networks

venue: arXiv preprint arXiv:1902.09574,

authors:
Xavier Glorot
Yoshua Bengio

title: Understanding the difficulty of training deep feedforward neural networks

venue: In Proceedings of the thirteenth international conference on artificial intelligence and statistics,

authors:
Xavier Glorot
Antoine Bordes
Yoshua Bengio

title: Deep sparse rectifier neural networks

venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,

authors:
Stephen José Hanson
Lorien Y Pratt

title: Comparing biases for minimal network construction with back-propagation

authors:
B. Hassibi
D.G. Stork
G.J. Wolff

title: Optimal brain surgeon and general network pruning

venue: In IEEE International Conference on Neural Networks, pp. 293–299

authors:
Babak Hassibi
David G. Stork

venue: In NIPS,

authors:
Babak Hassibi
David G. Stork
Stork Crc

title: Ricoh. Com. Second order derivatives for network pruning: Optimal brain surgeon

venue: In Advances in Neural Information Processing Systems

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

venue: In Proceedings of the IEEE international conference on computer vision,

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Geoffrey Hinton
Nitish Srivastava
Kevin Swersky

title: Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

venue: Cited on,

title: Untersuchungen zu dynamischen neuronalen netzen

venue: Diploma, Technische Universität München,

authors:
Sepp Hochreiter
Yoshua Bengio
Paolo Frasconi
Jürgen Schmidhuber

title: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies

title: The Hardware Lottery

authors:
Sara Hooker
Aaron Courville
Gregory Clark
Yann Dauphin
Andrea Frome

title: What Do Compressed Deep Neural Networks Forget? arXiv e-prints, art

authors:
Kurt Hornik
Maxwell Stinchcombe
Halbert White

title: Multilayer feedforward networks are universal approximators

title: computing’s energy problem (and what we can do about it)

venue: IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate

venue: shift. CoRR,

authors:
Yiding Jiang
Behnam Neyshabur
Hossein Mobahi
Dilip Krishnan
Samy Bengio

title: Fantastic generalization measures and where to find them

authors:
Xiaojie Jin
Chunyan Xu
Jiashi Feng
Yunchao Wei
Junjun Xiong
Shuicheng Yan

title: Deep learning with s-shaped rectified linear activation units

venue: arXiv preprint arXiv:1512.07030,

title: A simple procedure for pruning back-propagation trained neural networks

title: A new measure of rank correlation

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Alex Krizhevsky
Vinod Nair
Geoffrey Hinton

title: Cifar-10 and cifar-100 datasets

venue: URl: https://www. cs. toronto. edu/kriz/cifar. html,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing

authors:
Anders Krogh
John A Hertz

title: A simple weight decay can improve generalization

authors:
N.D. Lane
P. Warden

title: The deep (learning) transformation of mobile and embedded computing

authors:
Yann LeCun
John S. Denker
Sara A. Solla

title: Optimal Brain Damage

authors:
Namhoon Lee
Thalaiyasingam Ajanthan
Philip H.S. Torr

title: SNIP: single-shot network pruning based on connection sensitivity

authors:
Namhoon Lee
Thalaiyasingam Ajanthan
Philip HS Torr

title: Snip: Single-shot network pruning based on connection sensitivity

venue: arXiv preprint arXiv:1810.02340,

authors:
Namhoon Lee
Thalaiyasingam Ajanthan
Stephen Gould
Philip HS Torr

title: A signal propagation perspective for pruning neural networks at initialization

authors:
Tianlin Liu
Friedemann Zenke

title: Finding trainable sparse networks through neural tangent transfer

venue: arXiv preprint arXiv:2006.08228,

authors:
Zhuang Liu
Mingjie Sun
Tinghui Zhou
Gao Huang
Trevor Darrell

title: Rethinking the value of network pruning

venue: arXiv preprint arXiv:1810.05270,

title: Rethinking the Value of Network Pruning

authors:
Ilya Loshchilov
Frank Hutter

title: Decoupled weight decay regularization

venue: arXiv preprint arXiv:1711.05101,

authors:
C. Louizos
M. Welling
D.P. Kingma

title: Learning Sparse Neural Networks through L 0 Regularization

venue: ArXiv e-prints,

authors:
Zhou Lu
Hongming Pu
Feicheng Wang
Zhiqiang Hu
Liwei Wang

title: The expressive power of neural networks: A view from the width

title: Handbook of biological statistics, volume 2. sparky house publishing

authors:
Duane Merrill
Michael Garland

title: Merge-based parallel sparse matrix-vector multiplication

venue: In SC’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,

authors:
Decebal Constantin Mocanu
Elena Mocanu
Peter Stone
Phuong H Nguyen
Madeleine Gibescu
Antonio Liotta

title: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

venue: Nature communications,

authors:
Hesham Mostafa
Xin Wang

title: Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization

venue: arXiv preprint arXiv:1902.05967,

authors:
Michael C Mozer
Paul Smolensky

title: Skeletonization: A technique for trimming the fat from a network via relevance assessment

venue: Advances in Neural Information Processing Systems

authors:
Vinod Nair
Geoffrey E Hinton

title: Rectified linear units improve restricted boltzmann machines

venue: In ICML,

authors:
Sharan Narang
Erich Elsen
Gregory Diamos
Shubho Sengupta

title: Exploring Sparsity in Recurrent Neural Networks

title: Connectionist learning of belief networks

venue: Artificial intelligence,

authors:
Jorge Nocedal
Annick Sartenaer
Ciyou Zhu

title: On the behavior of the gradient norm in the steepest descent method

venue: Computational Optimization and Applications,

authors:
Razvan Pascanu
Tomas Mikolov
Yoshua Bengio

title: On the difficulty of training recurrent neural networks

venue: In International conference on machine learning,

title: Some methods of speeding up the convergence of iteration methods

venue: USSR Computational Mathematics and Mathematical Physics,

authors:
Prajit Ramachandran
Barret Zoph
Quoc V Le

title: Swish: a self-gated activation function

venue: arXiv preprint arXiv:1710.05941,

authors:
Herbert Robbins
Sutton Monro

title: A stochastic approximation method

venue: The annals of mathematical statistics,

title: An overview of gradient descent optimization algorithms

venue: arXiv preprint arXiv:1609.04747,

authors:
Ravi K Samala
Heang-Ping Chan
Lubomir M Hadjiiski
Mark A Helvie
Caleb Richter
Kenny Cha

title: Evolutionary pruning of transfer learned deep convolutional neural network for breast cancer diagnosis in digital breast tomosynthesis

venue: Physics in Medicine & Biology,

authors:
Abigail See
Minh-Thang Luong
Christopher D. Manning

title: Compression of Neural Machine Translation Models via Pruning

authors:
Ekdeep Singh Lubana
Robert P Dick

title: A gradient flow framework for analyzing network pruning

venue: arXiv e-prints, pp. arXiv–2009,

authors:
Emma Strubell
Ananya Ganesh
Andrew McCallum

title: Energy and policy considerations for deep learning in nlp, 2019

title: Sparse connection and pruning in large dynamic artificial neural networks

authors:
Ilya Sutskever
James Martens
George Dahl
Geoffrey Hinton

title: On the importance of initialization and momentum in deep learning

authors:
Hidenori Tanaka
Daniel Kunin
Daniel LK Yamins
Surya Ganguli

title: Pruning neural networks without any data by iteratively conserving synaptic flow

venue: arXiv preprint arXiv:2006.05467,

authors:
Neil C. Thompson
Kristjan Greenewald
Keeheon Lee
Gabriel F. Manso

title: The Computational Limits of Deep Learning

authors:
Chaoqi Wang
Guodong Zhang
Roger Grosse

title: Picking winning tickets before training by preserving gradient flow

venue: arXiv preprint arXiv:2002.07376,

authors:
P. Warden
D. Situnayake

title: TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers

venue: O’Reilly Media,

authors:
W. Wen
C. Wu
Y. Wang
Y. Chen
H. Li

title: Learning Structured Sparsity in Deep Neural Networks

title: Individual comparisons by ranking methods

venue: Biometrics Bulletin,

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

venue: arXiv preprint arXiv:1708.07747,

authors:
Chiyuan Zhang
Samy Bengio
Moritz Hardt
Benjamin Recht
Oriol Vinyals

title: Understanding deep learning requires rethinking generalization

venue: arXiv preprint arXiv:1611.03530,

authors:
Yue Zhao
Jiajia Li
Chunhua Liao
Xipeng Shen

title: Bridging the gap between deep learning and sparse matrix format selection

venue: In Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming,

authors:
M. Zhu
S. Gupta

title: To prune, or not to prune: exploring the efficacy of pruning for model compression

heading: 1 INTRODUCTION

text:
Over the last decade, a “bigger is better” race in the number of model parameters has gripped the field of machine learning (Amodei et al., 2018; Thompson et al., 2020), primarily driven by overparameterized deep neural networks (DNNs). Additional parameters improve top-line metrics, but drive up the cost of training (Horowitz, 2014; Strubell et al., 2019; Hooker, 2020) and increase the latency and memory footprint at inference time (Warden & Situnayake, 2019; Samala et al., 2018; Lane & Warden, 2018). Moreover, overparameterized networks have been shown to be more prone to memorization (Zhang et al., 2016).
To address some of these limitations, there has been a renewed focus on compression techniques that preserve top-line performance while improving efficiency. A large amount of research focus has centered on pruning, where weights estimated to be unnecessary are removed from the network at the end of training (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993a; Ström, 1997; Hassibi et al., 1993b; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017). Pruning has shown a remarkable ability to preserve top-line metrics of performance, even when removing the majority of weights (Hooker et al., 2019; Gale et al., 2019). However, most pruning techniques still require training a large, overparameterized model before pruning a subset of weights.
Due to the drawbacks of starting dense prior to introducing sparsity, there has been a recent focus on methods that allow networks which start sparse at initialization, to converge to similar performance as dense networks (Frankle & Carbin, 2018; Frankle et al., 2019b; Liu et al., 2018a). These efforts have focused disproportionately on trying to understand the properties of initial sparse weight distributions that allow for convergence. However, while this work has had some success, focusing on initialization alone has proven to be inadequate (Frankle et al., 2020; Evci et al., 2019).
In this work, we take a broader view of why training sparse networks to converge to the same performance as dense networks has proven to be elusive. We reconsider many of the basic building blocks of the training process and ask whether they disadvantage sparse networks or not. Our work focuses on the behaviour of networks with random, fixed sparsity at initialization and we aim to gain further intuition into how these networks learn. Furthermore, we provide tooling tailored to the analysis of these networks.
In order to effectively study sparse network optimization in a controlled environment, we propose an experimental framework, Same Capacity Sparse vs Dense Comparison (SC-SDC). Contrary to most prior work comparing sparse to dense networks, where overparameterized dense networks are compared to smaller sparse networks, SC-SDC compares sparse networks to their equivalent capacity dense networks (same number of active connections and depth). This ensures that the results are a direct result of sparse connections themselves and not due to having more or fewer weights (as is the case when comparing large, dense networks to smaller, sparse networks).
We go beyond simply comparing top-line metrics by also measuring the impact on gradient flow of each intervention. Historically, exploding and vanishing gradients were a common problem in neural networks (Hochreiter et al., 2001; Hochreiter, 1991; Bengio et al., 1994; Glorot & Bengio, 2010; Goodfellow et al., 2016). Recent work has suggested that poor gradient flow is an exacerbated issue in sparse networks (Wang et al., 2020; Evci et al., 2020). To accurately measure gradient flow in sparse networks, we propose a normalized measure of gradient flow, which we term Effective Gradient Flow (EGF) – this measure normalizes by the number of active weights and thus is better suited to studying the training dynamics of sparse networks. We use this measure in conjunction with SC-SDC, to see where sparse optimization fails and to consider where this failure could be a result of poor gradient flow.
Contributions Our contributions can be enumerated as follows:
1. Measuring effective gradient flow We conduct large scale experiments to evaluate the role of regularization, optimization and architecture choices on sparse models. We evaluate multiple datasets and architectures and propose a new measure of gradient flow, Effective Gradient Flow (EGF), that we show to be a stronger predictor of top-line metrics such as accuracy and loss than current gradient flow formulations.
2. Batch normalization plays a disproportionate role in stabilizing sparse networks We show that batch normalization is more important for sparse networks than it is for dense networks, which suggests that gradient instability is a key obstacle to starting sparse.
3. Not all optimizers and regulizers are created equal Weight decay and data augmentation can hurt sparse network optimization, particularly when used in conjunction with accelerating, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam (Kingma & Ba, 2014) and RMSProp (Hinton et al., 2012). We show this is highly correlated to a high EGF (gradient flow) and how batch normalization helps stabilize EGF.
4. Changing activation functions can benefit sparse networks We benchmark a wide set of activation functions, specifically ReLU (Nair & Hinton, 2010) and non-saturating activation functions such as PReLU (He et al., 2015), ELU (Clevert et al., 2015), SReLU (Jin et al., 2015),Swish (Ramachandran et al., 2017) and Sigmoid (Neal, 1992). Our results show that when using adaptive optimization methods, Swish is a promising activation function, while when using stochastic gradient descent, PReLU preforms better than the other activation functions.
Implications Our work is timely as sparse training dynamics are poorly understood. Most training algorithms and methods have been developed to suit training dense networks. Our work provides insight into the nature of sparse optimization and suggests a wider viewpoint beyond initialization is necessary to converge sparse networks to comparable performance as dense. Our proposed approach provides a more accurate measurement of the training dynamics of sparse networks and can be used to inform future work on the design of networks and optimization techniques that are tailored explicitly to sparsity.

heading: 2 METHODOLOGY

heading: 2.1 SAME CAPACITY SPARSE VS DENSE COMPARISON

text:
Our goal is to measure what architecture and optimization choices favor sparse networks relative to dense networks. To fairly compare sparse and dense networks, we propose Same Capacity Sparse vs Dense Comparison (SC-SDC), a simple framework which allows us to study sparse network optimization and identify what training configurations are not well suited for sparse networks.
SC-SDC can be summarized as follows (See Figure 1 for an overview):
1. Initialize For a chosen network depth (number of layers) L and a maximum network width NMaxW , we compare sparse and dense networks at various widths, while ensuring they have the same paramater count.
Initially, we mask the weights θS of sparse network S:
alS = θ l S ml , alD = θlD, for l = 1, . . . , L (1)
, where θlS ml denotes an element-wise product of the weights θS of layer l and the random binary matrix (mask) for layer l, ml, alS is the nonzero weights in layer l of sparse network S and a l D is the nonzero weights in layer l of dense network D (all the weights since no masking occurs).
For a fair comparison, we need to ensure the same number of nonzero weights for sparse network S and dense network D , across each layer L.
||alS ||0 = ||alD||0, for l = 1, . . . , L (2)
We provide more implementation details of how we achieve this in Appendix A.1.
2. Match active weight distributions Following prior work (Liu et al., 2018b; Gale et al., 2019), we ensure the nonzero weights at initialization of the sparse and dense networks are sampled from the same distribution at each layer as follows:
alS ∼ P l , alD ∼ P l, for l = 1, . . . , L (3) , where P l refers to the initial weight distribution at layer l, for example Kaiming initialization (He et al., 2015). This ensures that both sets of active weights (sparse and dense) are initially sampled from the same distribution.
3. Train We then train the sparse and dense networks for 1000 epochs (allowing for convergence).
4. Evaluate the better architecture We gather the results across the widths/capacity levels and conduct a paired, one-tail Wilcoxon signed-rank test (Wilcoxon, 1945) to evaluate the better architecture. Our null hypothesis (H0) is that sparse networks have similar or worse test accuracy than dense networks (lower or the same median), while our alternative hypothesis (H1) is that sparse networks have better test accuracy performance than dense networks of the same capacity (higher median). This can be formulated as:
H0 : Sparse <= Dense , H1 : Sparse > Dense (4)
Our goal of SC-SDC is to compare sparse and dense networks at the same capacity level. By same capacity, we are referring to the same number of active weights, but other notions of same capacity can also be used. We briefly discuss this in Appendix A.1.
We compare the average absolute Kendall Rank correlation between different formulations of gradient flow and generalization. The subscript denotes the p-norm (l1 or l2 norm). We see that EGF has higher absolute correlation when compared to standard gradient flow measures. We also see that is consistent across Fashion MNIST (see Appendix B.1).

heading: 2.2 MEASURING GRADIENT FLOW

text:
Gradient flow (GF) is used to study optimization dynamics and typically approximated by taking the norm of the gradients of the network (Pascanu et al., 2013; Nocedal et al., 2002; Chen et al., 2018; Wang et al., 2020; Evci et al., 2020).
We consider a feedforward neural network f : RD → R, with function inputs x ∈ RD and network weights θ. The gradient norm is usually computed by concatenating all the gradients of a network into a single vector, g = ∂C∂θ , whereC is our cost function. Then the vector norm is taken as follows:
gfp = ||g||p, (5)
where p denotes the pth-norm.
Effective Gradient Flow Traditional measures of gradient flow take the l1 or l2 norm of all the gradients (Chen et al., 2018; Pascanu et al., 2013; Evci et al., 2020). This is not appropriate for sparse networks, as this would include gradients of masked weights which have no influence on the forward pass. Furthermore, computing l1 or l2 across all weights in the networks gives disproportionate influence to layers with more weights. We instead propose a simple modification of Equation 5, which we term Effective Gradient Flow (EGF), that computes the average, masked gradient (only gradients of active weights) norm across all layers.
We calculate EGF as follows:
g = ( ∂C ∂θ1 m1, ∂C ∂θ2 m2, . . . , ∂C ∂θC mL) for l = 1, . . . , L (6)
egfp = ∑L n=1 ||gi||p L , (7)
where L is number of layers and ∂C ∂θl ml denotes an element-wise product of the gradients of layer l, ∂C ∂θl
, and the mask ml applied to the weights of layer l. For a fully dense network, ml is a matrix of all ones, since no gradients are masked.
EGF has the following favourable properties:
• Gradient flow is evenly distributed across layers EGF distributes the gradient norm across the layers equally, preventing layers with a lot of weights from dominating the measure and also preventing layers with vanishing gradients from being hidden in the formulation, as is the case with equation 5 (when all gradients are appended together).
• Only gradients of active weights are used EGF ensures that for sparse networks, only gradients of active weights are used. Even though weights are masked, their gradients are not necessarily
• Possibility for application in gradient-based pruning methods Tanaka et al. (2020) showed that gradient-based pruning methods like GRASP (Wang et al., 2020) and SNIP (Lee et al., 2018a), disproportionately prune large layers and are susceptible to layer-collapse, which is when an algorithm prunes all the weights in a specific layer. Due to the fact that EGF is evenly distributed across layers, maintaining EGF (as opposed to standard gradient norm) could possibly be used as a pruning criteria. Furthermore, current approaches measuring or approximating the change in gradient flow during pruning in sparse networks (Wang et al., 2020; Evci et al., 2020; Singh Lubana & Dick, 2020), could benefit from this new formulation.
To evaluate EGF against other standard gradient norm measures, such as the l1 and l2 norm, we empirically compare these measures and their correlation to test loss and accuracy. We take the absolute average of the Kendall Rank correlation (Kendall, 1938), across the different experiment configurations. We follow a similiar approach to Jiang et al. (2019), but unlike their work which has focused on correlating network complexity measures to the generalization gap, we measure the correlation of gradient flow to performance (accuracy and loss). We measure gradient flow at 10 points evenly spaced throughout training, specifically at the end of epoch 0, 99. 199, 299, 399, 499, 599, 699, 799, 899 and 999.
Our results from Table 1 shows that EGF has a higher average absolute correlation to both test loss and accuracy. This is also true of Fashion MNIST (see Appendix B.1). Due to the comparative benefits of EGF, we use it for the remainder of the paper to measure the impact of interventions. We include all measures of gradient flow in Appendix B for completeness.

heading: 2.3 ARCHITECTURE, NORMALIZATION, REGULARIZATION AND OPTIMIZER VARIANTS

text:
We briefly describe our key experiment variants below, and also include for completeness all unique variants in Table 2.
Activation functions ReLU networks (Nair & Hinton, 2010) are known to be more resilient to vanishing gradients than networks that use Sigmoid or Tanh activations, since they only result in vanishing gradients when the input is less than zero, while on active paths, due to ReLU’s linearity, the gradients flow uninhibited (Glorot et al., 2011). Although most experiments are run on ReLU networks, we also explore different activation functions, namely PReLU (He et al., 2015), ELU (Clevert et al., 2015), Swish (Ramachandran et al., 2017), SReLU (Jin et al., 2015) and Sigmoid (Neal, 1992).
Batch normalization and Skip Connections Other methods to help alleviate the vanishing gradient problem include the addition of skip connections (every two layers) (Srivastava et al., 2015; He et al., 2016) and batch normalization (Ioffe & Szegedy, 2015). We empirically explore these methods.
Optimization and Regularization techniques We explore the impact of popular regularization methods: weight decay/l2 regularization (0.0001) (Krogh & Hertz, 1992; Hanson & Pratt, 1989) and data augmentation (random crops and random horizontal flipping (Krizhevsky et al., 2012)). Furthermore, we benchmark the impact of the most widely used optimizers such as minibatch stochastic gradient descent (with momentum (0.9) (Sutskever et al., 2013; Polyak, 1964) and without momen-
tum (Robbins & Monro, 1951)) , Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSProp (Hinton et al., 2012).

heading: 3 EMPIRICAL SET-UP

text:
SC-SDC MLP Setting We use the SC-SDC empirical setting (section 2.1) for all experiment variants. We train over 6000 MLPs for 1000 epochs and evaluate performance on CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009). We compare sparse and dense networks across various widths, depths, learning rates, regularization and optimization methods as shown in Table 2.
We choose a max network width NMaxW of n+ 4, where n is the input dimension of the network. In the case of CIFAR, n = 3072 and so our maximum width NMaxW = 3076. We repeat these experiments with one, two and four hidden layers, with the number of active weights in these networks ranging from 949, 256 to 31, 765, 568 weights. In section 4, we discuss results achieved using four hidden layers on CIFAR-100 and we provide the one and two hidden layer results in Appendix C.
Dense Width Following from SC-SDC, these networks are compared at various network widths, specifically a width of 308, 923, 1538, 2153, 2768 (10%, 30%, 50%, 70% and 90% of our maximum width NMaxW (3076)) as shown in Table 2. We use the term dense width to refer to the width of a network if that network was dense. For example, when comparing sparse and dense networks at a dense width of 308, this means the dense network has a width of 308, while the sparse network has a width of NMaxW (3076), but has the same number of active connections as the dense counterpart. We provide more detailed discussion of the choices made in the SC-SDC implementation in Appendix A.1.
Extended CNN Setting We also extend our experiments to Wide Resnet-50 (the WRN-28-10 variant) (Zagoruyko & Komodakis, 2016) and use the optimization and regularization configurations from the paper.

heading: 4 RESULTS AND DISCUSSION

text:
4.1 COMPARISON OF DENSE AND SPARSE INTERVENTIONS USING SC-SDC
In this section, we use the results of the Wilcoxon signed rank test from SC-SDC to identify where optimization choices are currently well suited for sparse networks and which are not. Furthermore, for each variant, we also measure the gradient flow using EGF(egf2 (6))1 as described in the previous section. Our main findings show that:
1. Batch normalization is critical to training sparse networks, more so than it is for dense networks. This suggests that gradient instability is a key obstacle for sparse optimization.
2. Weight decay (with and without batch normalization) and data augmentation ( without batch normalization) can hurt both sparse and dense network optimization. This particularly occurs when using accelerated, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam and RMSProp (Ruder, 2016). In these methods, large EGF (gradient flow) strongly correlates to poor test accuracy.
3. Non-saturating activation functions, such as Swish (Ramachandran et al., 2017) and PReLU (He et al., 2015), achieve promising results in both the sparse and dense regime. However, these results are more statistically significant when sparse networks are used and so this could motivate for the use of similar activation functions for training sparse networks.
Batch normalization plays a disproportionate role in stabilizing sparse networks Batch normalization ensures that the distribution of the nonlinearity inputs remains stable as the network trains, which was hypothesized to help stabilize gradient propagation (gradients do not explode or vanish) (Ioffe & Szegedy, 2015). Following from Table 3a and 3b, we see that batch normalization is statistically more important for sparse network performance than it is for dense networks, across most configurations and learning rates.
1Note we measure EGF at 10 points throughout training and take the average

heading: NR - No Regularization, BN - Batchnorm, SC - Skip Connections, DA - Data Augmention and L2- weight decay.

text:
From Table 3a and Figure 10a, we see methods such as L2 and data augmentation usually favour dense networks (apart from Adam and RMSProp, when using data augmentation). However, with the addition of batch normalization (Table 3b, L2 to L2 BN and DA to DA BN ), these methods favour sparse variants. This is especially apparent in Figure 10a, where batch normalization improves performance across all sparse optimizers, while resulting in a lower, more stable EGF. This further emphasizes the importance of stabilizing gradient flow, particularly in sparse networks.
Weight Decay and Data Augmentation can hurt sparse network optimization When we take a closer look at the effects of weight decay and data augmentation on sparse network accuracy (Figure 2,Figure 13), we see that weight decay (even with batch normalization) drastically decreases accuracy when used with adaptive optimization methods that use an exponentially decaying average of past squared gradients (Adam and RMSProp). Furthermore, it results in distinctively larger EGF values, which hints at Adam and RMSProp being more sensitive to larger gradient norms than other optimizers. This agrees with Loshchilov & Hutter (2017), who proposed a different formulation of weight decay for adaptive methods, since the current L2 regularization formulation for adaptive methods could lead to weights with large gradients being regularized less, although this was not experimentally verified. In the context of data augmentation, we see poor test accuracy when it is used without batch normalization (Figure 10a). If used with batch normalization (Figure 2), it results in a lower EGF and best test accuracy. This further emphasized the need to stabilize gradient flow and how EGF can be used to this end.
The potential of non-saturating activation functions - Swish and PReLU We also explore the effect of different activation functions on sparse network optimization. For the activation function

text:
variants, the best configuration for each optimizer was chosen. For Adagrad, Adam and RMSProp we use BN, SC and DA, while for SGD, we use BN, SC, L2 and DA.
From Table 3c, we see that Swish, PReLU and Sigmoid favour sparse architectures, but from the performance results from Figure 12, we see that only Swish and PReLU are viable activation choices. We continue to see a consistent trend for adaptive methods (most notably in Adam and RMSProp), that higher EGF values, for example in SReLU, correspond to poor performance (Figure 11b), while promising methods result in a lower EGF value (such as Swish). This further emphasizes how EGF can be used to guide advances in network optimization.

heading: 4.2 GENERALIZATION OF RESULTS ACROSS ARCHITECTURE TYPES.

text: In this section, we move on from SC-SDC and extend our result to Wide ResNet-50. We note from Figure 3, that most of our results from SC-SDC also hold on larger, more complicated models. We see that L2 regularization (even with batch normalization) hurts performance for adaptive methods (Adagrad and Adam) and also results in higher EGF values (Figure 14). Furthermore, we also see data augmentation is beneficial when used with batch normalization. Finally, we see that Swish is a promising activation function for adaptive methods and leads to lower EGF (Figure 14). This shows that the SC-SDC results are not constrained to small scale experiments and that it can be used to learn about dynamics of larger, more complicated networks.

heading: 5 RELATED WORK

text:
Pruning at Initialization Methods that prune at initialization aim to start sparse, instead of first pre-training an overparameterized network and then pruning. These methods use certain criteria to estimate at initialization, which weights should remain active. This criteria includes using the connection sensitivity (Lee et al., 2018b), gradient flow (via the Hessian vector product) (Wang et al., 2020) and conversation of synaptic saliency (Tanaka et al., 2020). Another branch of pruning is Dynamic Sparse Training, which uses information gathered during the training process, to dynamically update the sparsity pattern of these sparse networks (Mostafa & Wang, 2019; Bellec et al., 2017; Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019; Evci et al., 2019). While our work is motivated by the same goal of allowing networks to start sparse and converge to the same performance as dense networks, we instead focus on the impact of optimization and regularization choices on sparse networks.
Sparse Network Optimization as Pruning Criteria Optimization in sparse networks has often been neglected in favour of studying network initialization. However, there has been work that has looked at sparse network optimization from different perspectives, mainly as a guide for pruning criteria. This includes using gradient information (Mozer & Smolensky, 1989; LeCun et al., 1989; Hassibi & Stork, 1992; Karnin, 1990), approximates of gradient flow (Wang et al., 2020; Dettmers & Zettlemoyer, 2019; Evci et al., 2020) and Neural Tangent Kernel (NTK) (Liu & Zenke, 2020) to guide the introduction of sparsity.
Sparse Network Optimization to study Network Dynamics Apart from use as pruning criteria, optimization information has been used to investigate aspects of sparse networks, such as their loss landscape (Evci et al., 2019), how they are impacted by SGD noise (Frankle et al., 2019a), the effect of different activation functions (Dubowski, 2020) and their weight initialization (Lee et al., 2019). Our work differs from these approaches as we consider more aspects of the optimization and regularization process in a controlled experimental setting (SC-SDC), while using EGF to reason about some of the results.

heading: 6 CONCLUSION AND FUTURE WORK

text: In this work, we take a wider view of sparse optimization strategies and introduce appropriate tooling to measure the impact of architecture and optimization choices on sparse networks (EGF , SC-SDC ). Our results show that weight decay and data augmentation can hurt optimization, when adaptive optimization methods are used and this usually corresponds to a much higher EGF.Furthermore, we show how batch normalization is critical to training sparse networks, more so than it is for dense networks as it helps stabilize gradient flow. We also show the potential of non-saturating activation functions for sparse networks such as Swish and PReLU. Finally, we show that our results extend to more complicated models like Wide ResNet-50.

heading: A SC-SDC

text:
In this section, we provide more information about SC-SDC and its benefits.
A.1 SC-SDC IMPLEMENTATION DETAILS
Wilcoxon Signed Rank Test This is a non-parametric test that compares dependent or paired samples, without assuming the differences in between the paired experiments are normally distributed (McDonald, 2009; Demšar, 2006).
Random Sparsity Our work focuses on the training dynamics of random, sparse networks. This ensures that what is learned is not dependent on a specific pruning method, but rather can be used to better understand sparse training in general. Going forward, it would be interesting to explore these dynamics on pruned networks.
We achieve random sparsity, by generating a random mask for each layer and then multiply the weights by this mask during each forward pass. The sparsity is distributed evenly across the network. For example, a 20% sparse MLP has 20% of the weights remaining in each layer.
Dense Width A critical component to how we specify our experiments is a term we define as dense width. In order to fairly compare sparse and dense networks, we need them to have the same number of active connections at each depth. In the case of sparse networks, this means ensuring they have the same number of active connections as the dense networks, while remaining sparse. Dense width refers to the width of a network if that network was dense. This process of comparing sparse and dense networks at different dense widths is illustrated in figure 5.
Fair comparison of Sparse and Dense networks As can be seen from figure 5, SC-SDC ensures the exact same active parameter count, but the sparse networks will be connected to more neurons. It is possible that the increased number of activations being used can lead to sparse networks having higher representational power, however most work on expressivity of neural networks looks at this from a depth perspective and proves certain depths of networks are universal approximators (Eldan & Shamir, 2016; Hornik et al., 1989; Funahashi, 1989).
To this end, we ensure these networks have the same depth, but we believe going forward an interesting direction would be ensuring they have a similar amount of active neurons.
SC-SDC comparison details For completeness, we provide more details of how we ensure sparse and dense networks are of the same capacity.
Following from equation 2, to ensure the same number of weights in sparse and dense networks, we can ensure they have the same number of active weights at each layer as follows:
||alS ||0 = ||alD||0, for l = 1, . . . , L (8)
This is achieved by masking each of the weight layers of sparse network S:
alS = θ l S ml for l = 1, . . . , L (9)
, where ml is a random binary matrix (mask) for layer l, s.t. ||ml||0 = alD, where alD is determined by the chosen capacity, these networks will be compared at.
For SC-SDC, we need a maximum network width NMaxW and comparison width NW . We choose a max network width NMaxW of n+ 4, where n is the input dimension of the network. In the case of CIFAR, n = 3072 and so our maximum width NMaxW = 3076. The choice of n + 4 follows from Lu et al. (2017), where the authors prove a universal approximation theorem for width-bounded ReLU networks, with width bounded to n + 4. Our comparison width, NW , is equivalent to dense widths we vary in our experiments - 308, 923, 1538, 2153, 2768.
The dimensions of each of layers are as follows:
1. First Layer:
θ1D ∈ RI×NW , θ1S ∈ RI×NMaxW , m1 ∈ {0, 1}I×NMaxW (10)
2. Intermediate Layers:
θ1S ∈ RNW×NW , θ1S ∈ RNMaxW×NMaxW , m{2,...,L−1} ∈ {0, 1}NMaxW×NMaxW (11)
3. Final Layer:
θLS ∈ RNW×O , θLS ∈ RNMaxW×O , mL ∈ {0, 1}NMaxW×O (12)
, where NMaxW is maximum width of the sparse layer, NW is the comparison width, I is the input dimension, O is output dimension, L is the number of layers in the network, θlS is the weights in layer l of sparse network S and θlD is the weights in layer l of dense network D.
This process would be the same for convolutional layers, but there would be a third dimension to handle the different channels. In figure 4, we provide an illustrative example showing how to ensure sparse and dense networks are compared fairly.
A.2 BENEFITS
The benefits of SC-SDC can be summarized as follows:
• We can better understand sparse network optimization. SC-SDC allows us to identify which optimization or regularization methods are poorly suited to sparse networks in a controlled setting, ensuring the results are a direct result of the sparse connections themselves.
• Learn at what parameter and size budget, sparse networks are better than dense. Comparing sparse and dense networks of the same capacity allows us to see which architecture is better at different configurations. In configurations where sparse architectures perform better, we could exploit advances in sparse matrix computation and storage (Zhao et al., 2018; Merrill & Garland, 2016) to simply default to sparse architectures.

heading: B GRADIENT FLOW

text:
B.1 EGF RESULTS ON FMNIST
We extend our experiments to Fashion MNIST (Xiao et al., 2017), a dataset that is distinctively different to the CIFAR datasets we used in section 2.2. We ran 450 experiments with networks with four hidden layers, using 0.001 as a learning rate and for 500 epochs. We varied configurations as follows:
• Optimizers - Adagrad, Adam and SGD with momentum. • Regularization methods - no regularization, batchnorm, skip connections, l2 (0.0001) and
data augmentation.
From table 4, we see that out of the gradient flow formulations, EGF still correlates better to generalization performance.
For completeness, we present the full set of results using the different formulations of gradient flow on CIFAR-100. Namely, we show ||g||1 (5) (Figure 6) ,||g||2 (5) (Figure 7) ,egf1 (6) (Figure 8) and egf2 (6) (Figure 9).

heading: C DETAILED RESULTS FOR SC-SDC

text:
In this section, we presented the detailed results for our experiments.
1. Detailed Results with a low learning rate (0.001). 2. Detailed Results with a high learning rate (0.1). 3. Results for different activation functions.
Adagrad 53.933 +/- 3.137 53.837 +/- 2.903 58.118 +/- 2.059 56.836 +/- 5.211 53.755 +/- 3.108 44.738 +/- 18.207 55.204 +/- 3.483 57.153 +/- 1.194 53.896 +/- 3.159 55.491 +/- 1.362 Adam 24.538 +/- 17.34 36.206 +/- 22.015 46.911 +/- 5.617 49.115 +/- 6.169 50.875 +/- 0.63 43.025 +/- 17.121 58.379 +/- 1.538 58.947 +/- 1.261 52.981 +/- 0.918 54.54 +/- 0.895 RMSProp 24.241 +/- 15.653 45.916 +/- 12.861 53.122 +/- 3.217 54.358 +/- 3.115 50.249 +/- 0.631 41.931 +/- 17.325 58.101 +/- 1.318 58.665 +/- 1.248 52.356 +/- 1.692 53.987 +/- 0.756 SGD 50.469 +/- 1.471 36.697 +/- 16.941 60.295 +/- 1.784 46.193 +/- 18.397 50.389 +/- 1.562 36.478 +/- 16.985 52.78 +/- 2.313 55.299 +/- 0.347 51.949 +/- 1.698 53.941 +/- 1.082 SGD with mom(0.9) 54.749 +/- 1.473 45.291 +/- 14.629 61.43 +/- 2.792 53.408 +/- 19.158 54.382 +/- 1.506 44.661 +/- 14.911 56.496 +/- 2.326 58.336 +/- 0.632 54.298 +/- 1.738 53.312 +/- 3.529
Adagrad 5.158 +/- 0.637 4.361 +/- 1.604 2.021 +/- 0.752 1.838 +/- 0.697 3.819 +/- 1.308 3.172 +/- 0.743 2.802 +/- 0.29 2.396 +/- 0.225 4.897 +/- 0.682 3.625 +/- 1.131 Adam 11.182 +/- 11.38 95.498 +/- 157.421 2.722 +/- 0.382 2.816 +/- 0.681 3.608 +/- 0.171 3.51 +/- 0.671 7.204 +/- 0.9 7.181 +/- 1.162 244.704 +/- 90.922 247.345 +/- 134.383 RMSProp 12.352 +/- 9.63 191.175 +/- 318.66 5.452 +/- 2.659 6.504 +/- 4.251 3.647 +/- 0.322 3.718 +/- 0.857 7.645 +/- 0.867 7.623 +/- 1.159 194.814 +/- 77.03 199.102 +/- 106.311 SGD 5.139 +/- 0.855 3.578 +/- 1.253 1.173 +/- 0.033 1.518 +/- 0.472 4.812 +/- 0.829 3.458 +/- 1.102 2.779 +/- 0.365 2.243 +/- 0.132 4.311 +/- 0.815 2.442 +/- 0.788 SGD with mom(0.9) 5.206 +/- 0.935 4.974 +/- 1.657 3.395 +/- 0.357 2.471 +/- 0.69 3.496 +/- 0.636 3.499 +/- 0.903 2.927 +/- 0.569 2.453 +/- 0.057 4.931 +/- 0.974 4.506 +/- 1.304
Adagrad 24.956 +/- 1.686 22.189 +/- 2.633 25.479 +/- 1.175 22.624 +/- 3.991 24.556 +/- 1.343 16.263 +/- 8.572 27.008 +/- 4.51 29.011 +/- 2.027 26.852 +/- 1.781 27.279 +/- 1.334 Adam 15.583 +/- 3.736 22.713 +/- 3.209 8.865 +/- 3.933 10.306 +/- 6.494 19.646 +/- 0.889 17.555 +/- 8.751 29.652 +/- 1.801 30.013 +/- 1.688 23.748 +/- 1.504 25.151 +/- 0.657 RMSProp 4.97 +/- 8.083 17.983 +/- 9.803 11.597 +/- 7.201 20.396 +/- 3.643 19.835 +/- 0.637 16.678 +/- 9.06 29.501 +/- 1.853 29.667 +/- 1.86 21.43 +/- 7.297 21.363 +/- 9.074 SGD 20.809 +/- 0.911 11.776 +/- 8.53 26.802 +/- 1.492 14.004 +/- 10.133 20.695 +/- 0.743 11.375 +/- 8.282 24.511 +/- 3.541 27.515 +/- 1.428 23.805 +/- 1.551 22.871 +/- 7.053 SGD with mom(0.9) 25.269 +/- 1.456 17.466 +/- 8.93 29.958 +/- 3.146 24.143 +/- 12.147 24.741 +/- 1.38 16.867 +/- 8.621 27.301 +/- 3.007 28.893 +/- 1.526 26.817 +/- 1.83 25.595 +/- 3.516
Adagrad 9.714 +/- 3.113 10.018 +/- 5.165 4.584 +/- 1.222 4.181 +/- 1.036 9.112 +/- 2.96 10.83 +/- 4.376 4.275 +/- 0.491 3.784 +/- 0.042 7.227 +/- 2.173 6.592 +/- 2.377 Adam 209.854 +/- 400.362 10867.689 +/- 10783.68 5.044 +/- 0.288 8.551 +/- 3.795 32.786 +/- 12.029 15.217 +/- 12.419 19.798 +/- 6.003 20.605 +/- 6.057 779.235 +/- 319.831 715.685 +/- 395.333 RMSProp 222.053 +/- 450.311 3680.784 +/- 5265.27 14.482 +/- 16.75 5.609 +/- 0.51 22.649 +/- 5.115 15.09 +/- 9.165 20.313 +/- 5.053 21.787 +/- 3.96 649.551 +/- 366.815 443.976 +/- 348.581 SGD 10.394 +/- 1.219 5.936 +/- 2.56 3.082 +/- 0.064 3.786 +/- 0.619 10.222 +/- 1.053 5.961 +/- 2.521 4.286 +/- 0.652 3.704 +/- 0.128 8.212 +/- 0.61 4.608 +/- 1.706 SGD with mom(0.9) 13.425 +/- 2.123 12.899 +/- 4.883 7.215 +/- 0.572 5.239 +/- 1.298 9.114 +/- 1.778 9.882 +/- 3.843 4.724 +/- 0.861 4.135 +/- 0.203 10.16 +/- 1.981 8.783 +/- 0.955
Adagrad 19.469 +/- 10.563 31.377 +/- 15.801 22.62 +/- 16.083 37.885 +/- 17.664 19.739 +/- 12.699 40.133 +/- 14.52 56.194 +/- 1.319 57.536 +/- 0.712 15.615 +/- 9.853 30.756 +/- 17.113 Adam 10.0 +/- 0.0 10.0 +/- 0.005 10.0 +/- 0.0 10.0 +/- 0.0 9.998 +/- 0.012 9.98 +/- 0.036 53.191 +/- 2.687 54.964 +/- 1.88 9.999 +/- 0.003 10.001 +/- 0.005 RMSProp 9.999 +/- 0.003 10.001 +/- 0.01 10.0 +/- 0.0 10.001 +/- 0.003 10.0 +/- 0.0 10.191 +/- 0.578 53.328 +/- 1.022 53.62 +/- 1.056 10.0 +/- 0.004 10.001 +/- 0.011 SGD 57.603 +/- 1.546 56.487 +/- 3.866 51.81 +/- 6.396 60.065 +/- 1.465 8.255 +/- 19.659 30.631 +/- 26.518 59.191 +/- 1.805 60.295 +/- 0.949 57.599 +/- 1.302 56.398 +/- 2.893 SGD with mom(0.9) 9.303 +/- 2.608 12.872 +/- 14.06 10.0 +/- 0.0 19.086 +/- 18.81 10.579 +/- 10.646 35.2 +/- 23.74 57.822 +/- 1.012 59.058 +/- 0.886 9.334 +/- 2.582 9.117 +/- 9.037
Adagrad 4.332 +/- 3.705 6.299 +/- 3.469 2.036 +/- 0.344 1.895 +/- 0.263 3.21 +/- 1.888 3.186 +/- 1.288 5.979 +/- 0.403 5.19 +/- 1.109 3.934 +/- 2.999 5.788 +/- 3.694 Adam 2.328 +/- 0.066 6.323 +/- 11.003 2.303 +/- 0.0 2.303 +/- 0.0 405749.405 +/- 1051430.43 343515.597 +/- 1330067.381 51.96 +/- 34.783 60.026 +/- 18.807 4.174 +/- 7.207 2.366 +/- 0.17 RMSProp 1339.999 +/- 5147.876 275.11 +/- 831.374 2.303 +/- 0.0 2.303 +/- 0.002 212.06 +/- 812.388 43851.25 +/- 122539.555 40.846 +/- 17.869 107.835 +/- 42.366 30.677 +/- 104.926 349.709 +/- 1315.569 SGD 6.609 +/- 1.155 7.01 +/- 2.525 4.964 +/- 1.373 3.744 +/- 1.348 0.339 +/- 0.897 1.659 +/- 1.237 3.758 +/- 0.389 3.254 +/- 0.341 4.634 +/- 0.791 4.674 +/- 1.132 SGD with mom(0.9) 2.138 +/- 0.615 3.166 +/- 4.273 2.303 +/- 0.0 2.129 +/- 0.359 1.797 +/- 0.975 2.175 +/- 0.895 6.208 +/- 0.244 5.396 +/- 0.905 2.149 +/- 0.595 4.388 +/- 8.928
Adagrad 4.832 +/- 3.339 11.731 +/- 7.078 10.304 +/- 7.42 14.827 +/- 8.334 5.383 +/- 5.543 14.899 +/- 9.16 28.804 +/- 1.905 29.826 +/- 1.857 5.885 +/- 3.829 11.649 +/- 6.443 Adam 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 0.999 +/- 0.007 22.995 +/- 1.308 23.792 +/- 2.673 1.0 +/- 0.0 1.001 +/- 0.003 RMSProp 1.0 +/- 0.0 1.001 +/- 0.004 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.008 24.424 +/- 1.281 25.13 +/- 1.514 0.999 +/- 0.003 0.999 +/- 0.005 SGD 25.719 +/- 11.926 23.833 +/- 11.53 11.46 +/- 5.91 22.79 +/- 10.489 28.546 +/- 8.056 24.077 +/- 11.917 30.83 +/- 2.898 31.749 +/- 1.631 26.638 +/- 9.891 28.838 +/- 3.675 SGD with mom(0.9) 1.0 +/- 0.0 5.53 +/- 6.788 1.0 +/- 0.0 8.005 +/- 9.069 4.471 +/- 5.342 12.283 +/- 10.29 31.041 +/- 2.264 31.778 +/- 1.709 1.003 +/- 0.009 1.689 +/- 2.477
Adagrad 7.667 +/- 4.947 14.836 +/- 8.468 4.13 +/- 0.358 5.685 +/- 2.939 6.024 +/- 3.425 6.571 +/- 2.298 9.474 +/- 0.926 8.717 +/- 1.854 9.247 +/- 5.927 15.916 +/- 9.915 Adam 4.664 +/- 0.201 8.052 +/- 8.361 4.605 +/- 0.0 4.605 +/- 0.0 27.215 +/- 87.569 64.341 +/- 137.011 170.973 +/- 84.218 367.302 +/- 141.566 4.631 +/- 0.081 4.752 +/- 0.508 RMSProp 11.792 +/- 23.242 57.154 +/- 202.471 4.605 +/- 0.0 4.606 +/- 0.003 4.605 +/- 0.0 5404.823 +/- 14437.166 146.552 +/- 73.377 748.996 +/- 361.425 5734.776 +/- 20650.808 5.919 +/- 3.307 SGD 10.48 +/- 2.908 11.058 +/- 3.433 9.797 +/- 5.488 11.97 +/- 5.69 5.7 +/- 1.647 5.322 +/- 0.59 5.587 +/- 0.934 5.004 +/- 0.265 11.017 +/- 7.566 8.611 +/- 1.653 SGD with mom(0.9) 4.605 +/- 0.0 12.49 +/- 11.831 4.605 +/- 0.0 4.367 +/- 0.342 3.903 +/- 1.591 4.965 +/- 0.518 10.612 +/- 1.41 9.014 +/- 2.289 4.607 +/- 0.005 16.577 +/- 29.665