summaries: This paper proposes effective gradient flow (EGF), which is a layer-wise normalized gradient flow. Compared to (unnormalized) gradient flow, the paper shows that the proposed EGF is (slightly) better correlated with metrics like test loss and test accuracy (see Table 1). Given that this claim is supported with experimental results, the paper would become much stronger if a larger number and a more diverse set of data-sets were used (in addition to CIFAR-10 and CIFAR-100, which are two very similar image-data-sets) as to show that the claim holds more generally. Apart from that, given that the correlations are (only) about 0.4 in Table 1, it seems that only some aspects are explained by EGF.  Recently, initialization has been found critical to the model accuracy attained by sparse training [1]. In this paper, the authors studies the impact of factors other than initialization on the model accuracy attained by sparse training relative to dense training (under the same model parameter count). At the core of this paper, the authors argue that the effective gradient flow (grad norm from only activate model weight dimensions) is an effective indicator on the model accuracy attained by sparse training. Firstly, the authors show that the effective gradient flow attains higher correlation with model accuracy than the norm of full gradient (including gradients on sparsified weight dimensions) in sparse training. Secondly, the paper empirically demonstrate that in sparse training, 1) weight decay and data augmentation can hurt model accuracy, 2) batch normalization plays significantly role for model accuracy in sparse training and 3) non-saturating activations boost the magnitude of effective gradient flow and consequently improve model accuracy.  This paper looks at how choices in optimization affect how well you can train sparse networks. The authors come up with a new measure of effective gradient flow, which is important for good performance. They also compare sparse vs. dense networks across various optimizers, hyperparameters and activation functions, and find that batch norm and certain activation functions are beneficial for sparse networks
abstractText: Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider various choices made during training that might disadvantage sparse networks. We measure the gradient flow across different networks and datasets, and show that the default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks. Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results.
references.authors: Guillaume Bellec David Kappel Wolfgang Maass Robert Legenstein
references.title: Deep rewiring: Training very sparse deep networks
references.venue: arXiv preprint arXiv:1711.05136,
references.authors: Yoshua Bengio Patrice Simard Paolo Frasconi
references.title: Learning long-term dependencies with gradient descent is difficult
references.venue: IEEE transactions on neural networks,
references.authors: Zhao Chen Vijay Badrinarayanan Chen-Yu Lee Andrew Rabinovich
references.title: Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks
references.venue: In International Conference on Machine Learning,
references.authors: Djork-Arné Clevert Thomas Unterthiner Sepp Hochreiter
references.title: Fast and accurate deep network learning by exponential linear units (elus)
references.venue: arXiv preprint arXiv:1511.07289,
references.authors: Yann Le Cun John S. Denker Sara A. Solla
references.title: Optimal brain damage
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Janez Demšar
references.title: Statistical comparisons of classifiers over multiple data sets
references.venue: Journal of Machine learning research,
references.authors: Tim Dettmers Luke Zettlemoyer
references.title: Sparse networks from scratch: Faster training without losing performance
references.venue: arXiv preprint arXiv:1907.04840,
references.authors: Adam Dubowski
references.title: Activation function impact on sparse neural networks
references.venue: B.S. thesis, University of Twente,
references.authors: John Duchi Elad Hazan Yoram Singer
references.title: Adaptive subgradient methods for online learning and stochastic optimization
references.venue: Journal of machine learning research,
references.authors: Ronen Eldan Ohad Shamir
references.title: The power of depth for feedforward neural networks
references.venue: In Conference on learning theory,
references.authors: Utku Evci Trevor Gale Jacob Menick Pablo Samuel Castro Erich Elsen
references.title: Rigging the Lottery: Making All Tickets Winners
references.venue: arXiv e-prints, art
references.authors: Utku Evci Fabian Pedregosa Aidan Gomez Erich Elsen
references.title: The difficulty of training sparse neural networks
references.venue: arXiv preprint arXiv:1906.10732,
references.authors: Utku Evci Yani A Ioannou Cem Keskin Yann Dauphin
references.title: Gradient flow in sparse neural networks and how lottery tickets win
references.venue: arXiv preprint arXiv:2010.03533,
references.authors: Jonathan Frankle Michael Carbin
references.title: The lottery ticket hypothesis: Finding sparse, trainable neural networks
references.venue: arXiv preprint arXiv:1803.03635,
references.authors: Jonathan Frankle Gintare Karolina Dziugaite Daniel M Roy Michael Carbin
references.title: Linear mode connectivity and the lottery ticket hypothesis
references.venue: arXiv preprint arXiv:1912.05671,
references.authors: Jonathan Frankle Gintare Karolina Dziugaite Daniel M Roy Michael Carbin
references.title: Stabilizing the lottery ticket hypothesis
references.venue: arXiv preprint arXiv:1903.01611,
references.authors: Jonathan Frankle Gintare Karolina Dziugaite Daniel M. Roy Michael Carbin
references.title: Pruning neural networks at initialization: Why are we missing the mark
references.authors: Ken-Ichi Funahashi
references.title: On the approximate realization of continuous mappings by neural networks
references.venue: Neural networks,
references.authors: Trevor Gale Erich Elsen Sara Hooker
references.title: The State of Sparsity in Deep Neural Networks
references.venue: arXiv e-prints, art
references.authors: Trevor Gale Erich Elsen Sara Hooker
references.title: The state of sparsity in deep neural networks
references.venue: arXiv preprint arXiv:1902.09574,
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks
references.venue: In Proceedings of the thirteenth international conference on artificial intelligence and statistics,
references.authors: Xavier Glorot Antoine Bordes Yoshua Bengio
references.title: Deep sparse rectifier neural networks
references.venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,
references.authors: Stephen José Hanson Lorien Y Pratt
references.title: Comparing biases for minimal network construction with back-propagation
references.venue: In Advances in neural information processing systems,
references.authors: B. Hassibi D.G. Stork G.J. Wolff
references.title: Optimal brain surgeon and general network pruning
references.venue: In IEEE International Conference on Neural Networks, pp. 293–299
references.authors: Babak Hassibi David G. Stork
references.title: Second order derivatives for network pruning: Optimal brain surgeon
references.venue: In NIPS,
references.authors: Babak Hassibi David G. Stork Stork Crc
references.title: Ricoh. Com. Second order derivatives for network pruning: Optimal brain surgeon
references.venue: In Advances in Neural Information Processing Systems
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Geoffrey Hinton Nitish Srivastava Kevin Swersky
references.title: Neural networks for machine learning lecture 6a overview of mini-batch gradient descent
references.venue: Cited on,
references.authors: Sepp Hochreiter
references.title: Untersuchungen zu dynamischen neuronalen netzen
references.venue: Diploma, Technische Universität München,
references.authors: Sepp Hochreiter Yoshua Bengio Paolo Frasconi Jürgen Schmidhuber
references.title: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies
references.authors: Sara Hooker
references.title: The Hardware Lottery
references.venue: arXiv e-prints, art
references.authors: Sara Hooker Aaron Courville Gregory Clark Yann Dauphin Andrea Frome
references.title: What Do Compressed Deep Neural Networks Forget? arXiv e-prints, art
references.authors: Kurt Hornik Maxwell Stinchcombe Halbert White
references.title: Multilayer feedforward networks are universal approximators
references.venue: Neural networks,
references.authors: M. Horowitz
references.title: computing’s energy problem (and what we can do about it)
references.venue: IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate
references.venue: shift. CoRR,
references.authors: Yiding Jiang Behnam Neyshabur Hossein Mobahi Dilip Krishnan Samy Bengio
references.title: Fantastic generalization measures and where to find them
references.authors: Xiaojie Jin Chunyan Xu Jiashi Feng Yunchao Wei Junjun Xiong Shuicheng Yan
references.title: Deep learning with s-shaped rectified linear activation units
references.venue: arXiv preprint arXiv:1512.07030,
references.authors: Ehud D Karnin
references.title: A simple procedure for pruning back-propagation trained neural networks
references.venue: IEEE transactions on neural networks,
references.authors: Maurice G Kendall
references.title: A new measure of rank correlation
references.venue: Biometrika, 30(1/2):81–93,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Alex Krizhevsky Vinod Nair Geoffrey Hinton
references.title: Cifar-10 and cifar-100 datasets
references.venue: URl: https://www. cs. toronto. edu/kriz/cifar. html,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing
references.authors: Anders Krogh John A Hertz
references.title: A simple weight decay can improve generalization
references.venue: In Advances in neural information processing systems,
references.authors: N.D. Lane P. Warden
references.title: The deep (learning) transformation of mobile and embedded computing
references.authors: Yann LeCun John S. Denker Sara A. Solla
references.title: Optimal Brain Damage
references.venue: In NIPS,
references.authors: Namhoon Lee Thalaiyasingam Ajanthan Philip H.S. Torr
references.title: SNIP: single-shot network pruning based on connection sensitivity
references.venue: CoRR, abs/1810.02340,
references.authors: Namhoon Lee Thalaiyasingam Ajanthan Philip HS Torr
references.title: Snip: Single-shot network pruning based on connection sensitivity
references.venue: arXiv preprint arXiv:1810.02340,
references.authors: Namhoon Lee Thalaiyasingam Ajanthan Stephen Gould Philip HS Torr
references.title: A signal propagation perspective for pruning neural networks at initialization
references.authors: Tianlin Liu Friedemann Zenke
references.title: Finding trainable sparse networks through neural tangent transfer
references.venue: arXiv preprint arXiv:2006.08228,
references.authors: Zhuang Liu Mingjie Sun Tinghui Zhou Gao Huang Trevor Darrell
references.title: Rethinking the value of network pruning
references.venue: arXiv preprint arXiv:1810.05270,
references.authors: Zhuang Liu Mingjie Sun Tinghui Zhou Gao Huang Trevor Darrell
references.title: Rethinking the Value of Network Pruning
references.venue: CoRR, abs/1810.05270,
references.authors: Ilya Loshchilov Frank Hutter
references.title: Decoupled weight decay regularization
references.venue: arXiv preprint arXiv:1711.05101,
references.authors: C. Louizos M. Welling D.P. Kingma
references.title: Learning Sparse Neural Networks through L 0 Regularization
references.venue: ArXiv e-prints,
references.authors: Zhou Lu Hongming Pu Feicheng Wang Zhiqiang Hu Liwei Wang
references.title: The expressive power of neural networks: A view from the width
references.venue: In Advances in neural information processing systems,
references.authors: John H McDonald
references.title: Handbook of biological statistics, volume 2. sparky house publishing
references.authors: Duane Merrill Michael Garland
references.title: Merge-based parallel sparse matrix-vector multiplication
references.venue: In SC’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,
references.authors: Decebal Constantin Mocanu Elena Mocanu Peter Stone Phuong H Nguyen Madeleine Gibescu Antonio Liotta
references.title: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
references.venue: Nature communications,
references.authors: Hesham Mostafa Xin Wang
references.title: Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
references.venue: arXiv preprint arXiv:1902.05967,
references.authors: Michael C Mozer Paul Smolensky
references.title: Skeletonization: A technique for trimming the fat from a network via relevance assessment
references.venue: Advances in Neural Information Processing Systems
references.authors: Vinod Nair Geoffrey E Hinton
references.title: Rectified linear units improve restricted boltzmann machines
references.venue: In ICML,
references.authors: Sharan Narang Erich Elsen Gregory Diamos Shubho Sengupta
references.title: Exploring Sparsity in Recurrent Neural Networks
references.venue: arXiv e-prints, art
references.authors: Radford M Neal
references.title: Connectionist learning of belief networks
references.venue: Artificial intelligence,
references.authors: Jorge Nocedal Annick Sartenaer Ciyou Zhu
references.title: On the behavior of the gradient norm in the steepest descent method
references.venue: Computational Optimization and Applications,
references.authors: Razvan Pascanu Tomas Mikolov Yoshua Bengio
references.title: On the difficulty of training recurrent neural networks
references.venue: In International conference on machine learning,
references.authors: Boris T Polyak
references.title: Some methods of speeding up the convergence of iteration methods
references.venue: USSR Computational Mathematics and Mathematical Physics,
references.authors: Prajit Ramachandran Barret Zoph Quoc V Le
references.title: Swish: a self-gated activation function
references.venue: arXiv preprint arXiv:1710.05941,
references.authors: Herbert Robbins Sutton Monro
references.title: A stochastic approximation method
references.venue: The annals of mathematical statistics,
references.authors: Sebastian Ruder
references.title: An overview of gradient descent optimization algorithms
references.venue: arXiv preprint arXiv:1609.04747,
references.authors: Ravi K Samala Heang-Ping Chan Lubomir M Hadjiiski Mark A Helvie Caleb Richter Kenny Cha
references.title: Evolutionary pruning of transfer learned deep convolutional neural network for breast cancer diagnosis in digital breast tomosynthesis
references.venue: Physics in Medicine & Biology,
references.authors: Abigail See Minh-Thang Luong Christopher D. Manning
references.title: Compression of Neural Machine Translation Models via Pruning
references.venue: arXiv e-prints, art
references.authors: Ekdeep Singh Lubana Robert P Dick
references.title: A gradient flow framework for analyzing network pruning
references.venue: arXiv e-prints, pp. arXiv–2009,
references.authors: Emma Strubell Ananya Ganesh Andrew McCallum
references.title: Energy and policy considerations for deep learning in nlp, 2019
references.authors: Nikko Ström
references.title: Sparse connection and pruning in large dynamic artificial neural networks
references.authors: Ilya Sutskever James Martens George Dahl Geoffrey Hinton
references.title: On the importance of initialization and momentum in deep learning
references.venue: In International conference on machine learning,
references.authors: Hidenori Tanaka Daniel Kunin Daniel LK Yamins Surya Ganguli
references.title: Pruning neural networks without any data by iteratively conserving synaptic flow
references.venue: arXiv preprint arXiv:2006.05467,
references.authors: Neil C. Thompson Kristjan Greenewald Keeheon Lee Gabriel F. Manso
references.title: The Computational Limits of Deep Learning
references.venue: arXiv e-prints, art
references.authors: Chaoqi Wang Guodong Zhang Roger Grosse
references.title: Picking winning tickets before training by preserving gradient flow
references.venue: arXiv preprint arXiv:2002.07376,
references.authors: P. Warden D. Situnayake
references.title: TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers
references.venue: O’Reilly Media,
references.authors: W. Wen C. Wu Y. Wang Y. Chen H. Li
references.title: Learning Structured Sparsity in Deep Neural Networks
references.authors: Frank Wilcoxon
references.title: Individual comparisons by ranking methods
references.venue: Biometrics Bulletin,
references.authors: Han Xiao Kashif Rasul Roland Vollgraf
references.title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
references.venue: arXiv preprint arXiv:1708.07747,
references.authors: Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals
references.title: Understanding deep learning requires rethinking generalization
references.venue: arXiv preprint arXiv:1611.03530,
references.authors: Yue Zhao Jiajia Li Chunhua Liao Xipeng Shen
references.title: Bridging the gap between deep learning and sparse matrix format selection
references.venue: In Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming,
references.authors: M. Zhu S. Gupta
references.title: To prune, or not to prune: exploring the efficacy of pruning for model compression
sections.text: Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider various choices made during training that might disadvantage sparse networks. We measure the gradient flow across different networks and datasets, and show that the default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks. Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results.
sections.heading: 1 INTRODUCTION
sections.text: Over the last decade, a “bigger is better” race in the number of model parameters has gripped the field of machine learning (Amodei et al., 2018; Thompson et al., 2020), primarily driven by overparameterized deep neural networks (DNNs). Additional parameters improve top-line metrics, but drive up the cost of training (Horowitz, 2014; Strubell et al., 2019; Hooker, 2020) and increase the latency and memory footprint at inference time (Warden & Situnayake, 2019; Samala et al., 2018; Lane & Warden, 2018). Moreover, overparameterized networks have been shown to be more prone to memorization (Zhang et al., 2016).
  To address some of these limitations, there has been a renewed focus on compression techniques that preserve top-line performance while improving efficiency. A large amount of research focus has centered on pruning, where weights estimated to be unnecessary are removed from the network at the end of training (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993a; Ström, 1997; Hassibi et al., 1993b; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017). Pruning has shown a remarkable ability to preserve top-line metrics of performance, even when removing the majority of weights (Hooker et al., 2019; Gale et al., 2019). However, most pruning techniques still require training a large, overparameterized model before pruning a subset of weights.
  Due to the drawbacks of starting dense prior to introducing sparsity, there has been a recent focus on methods that allow networks which start sparse at initialization, to converge to similar performance as dense networks (Frankle & Carbin, 2018; Frankle et al., 2019b; Liu et al., 2018a). These efforts have focused disproportionately on trying to understand the properties of initial sparse weight distributions that allow for convergence. However, while this work has had some success, focusing on initialization alone has proven to be inadequate (Frankle et al., 2020; Evci et al., 2019).
  In this work, we take a broader view of why training sparse networks to converge to the same performance as dense networks has proven to be elusive. We reconsider many of the basic building blocks of the training process and ask whether they disadvantage sparse networks or not. Our work focuses on the behaviour of networks with random, fixed sparsity at initialization and we aim to gain further intuition into how these networks learn. Furthermore, we provide tooling tailored to the analysis of these networks.
  In order to effectively study sparse network optimization in a controlled environment, we propose an experimental framework, Same Capacity Sparse vs Dense Comparison (SC-SDC). Contrary to most prior work comparing sparse to dense networks, where overparameterized dense networks are compared to smaller sparse networks, SC-SDC compares sparse networks to their equivalent capacity dense networks (same number of active connections and depth). This ensures that the results are a direct result of sparse connections themselves and not due to having more or fewer weights (as is the case when comparing large, dense networks to smaller, sparse networks).
  We go beyond simply comparing top-line metrics by also measuring the impact on gradient flow of each intervention. Historically, exploding and vanishing gradients were a common problem in neural networks (Hochreiter et al., 2001; Hochreiter, 1991; Bengio et al., 1994; Glorot & Bengio, 2010; Goodfellow et al., 2016). Recent work has suggested that poor gradient flow is an exacerbated issue in sparse networks (Wang et al., 2020; Evci et al., 2020). To accurately measure gradient flow in sparse networks, we propose a normalized measure of gradient flow, which we term Effective Gradient Flow (EGF) – this measure normalizes by the number of active weights and thus is better suited to studying the training dynamics of sparse networks. We use this measure in conjunction with SC-SDC, to see where sparse optimization fails and to consider where this failure could be a result of poor gradient flow.
  Contributions Our contributions can be enumerated as follows:
  1. Measuring effective gradient flow We conduct large scale experiments to evaluate the role of regularization, optimization and architecture choices on sparse models. We evaluate multiple datasets and architectures and propose a new measure of gradient flow, Effective Gradient Flow (EGF), that we show to be a stronger predictor of top-line metrics such as accuracy and loss than current gradient flow formulations.
  2. Batch normalization plays a disproportionate role in stabilizing sparse networks We show that batch normalization is more important for sparse networks than it is for dense networks, which suggests that gradient instability is a key obstacle to starting sparse.
  3. Not all optimizers and regulizers are created equal Weight decay and data augmentation can hurt sparse network optimization, particularly when used in conjunction with accelerating, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam (Kingma & Ba, 2014) and RMSProp (Hinton et al., 2012). We show this is highly correlated to a high EGF (gradient flow) and how batch normalization helps stabilize EGF.
  4. Changing activation functions can benefit sparse networks We benchmark a wide set of activation functions, specifically ReLU (Nair & Hinton, 2010) and non-saturating activation functions such as PReLU (He et al., 2015), ELU (Clevert et al., 2015), SReLU (Jin et al., 2015),Swish (Ramachandran et al., 2017) and Sigmoid (Neal, 1992). Our results show that when using adaptive optimization methods, Swish is a promising activation function, while when using stochastic gradient descent, PReLU preforms better than the other activation functions.
  Implications Our work is timely as sparse training dynamics are poorly understood. Most training algorithms and methods have been developed to suit training dense networks. Our work provides insight into the nature of sparse optimization and suggests a wider viewpoint beyond initialization is necessary to converge sparse networks to comparable performance as dense. Our proposed approach provides a more accurate measurement of the training dynamics of sparse networks and can be used to inform future work on the design of networks and optimization techniques that are tailored explicitly to sparsity.
sections.heading: 2 METHODOLOGY
sections.heading: 2.1 SAME CAPACITY SPARSE VS DENSE COMPARISON
sections.text: Our goal is to measure what architecture and optimization choices favor sparse networks relative to dense networks. To fairly compare sparse and dense networks, we propose Same Capacity Sparse vs Dense Comparison (SC-SDC), a simple framework which allows us to study sparse network optimization and identify what training configurations are not well suited for sparse networks.
  SC-SDC can be summarized as follows (See Figure 1 for an overview):
  1. Initialize For a chosen network depth (number of layers) L and a maximum network width NMaxW , we compare sparse and dense networks at various widths, while ensuring they have the same paramater count.
  Initially, we mask the weights θS of sparse network S:
  alS = θ l S ml , alD = θlD, for l = 1, . . . , L (1)
  , where θlS ml denotes an element-wise product of the weights θS of layer l and the random binary matrix (mask) for layer l, ml, alS is the nonzero weights in layer l of sparse network S and a l D is the nonzero weights in layer l of dense network D (all the weights since no masking occurs).
  For a fair comparison, we need to ensure the same number of nonzero weights for sparse network S and dense network D , across each layer L.
  ||alS ||0 = ||alD||0, for l = 1, . . . , L (2)
  We provide more implementation details of how we achieve this in Appendix A.1.
  2. Match active weight distributions Following prior work (Liu et al., 2018b; Gale et al., 2019), we ensure the nonzero weights at initialization of the sparse and dense networks are sampled from the same distribution at each layer as follows:
  alS ∼ P l , alD ∼ P l, for l = 1, . . . , L (3) , where P l refers to the initial weight distribution at layer l, for example Kaiming initialization (He et al., 2015). This ensures that both sets of active weights (sparse and dense) are initially sampled from the same distribution.
  3. Train We then train the sparse and dense networks for 1000 epochs (allowing for convergence).
  4. Evaluate the better architecture We gather the results across the widths/capacity levels and conduct a paired, one-tail Wilcoxon signed-rank test (Wilcoxon, 1945) to evaluate the better architecture. Our null hypothesis (H0) is that sparse networks have similar or worse test accuracy than dense networks (lower or the same median), while our alternative hypothesis (H1) is that sparse networks have better test accuracy performance than dense networks of the same capacity (higher median). This can be formulated as:
  H0 : Sparse <= Dense , H1 : Sparse > Dense (4)
  Our goal of SC-SDC is to compare sparse and dense networks at the same capacity level. By same capacity, we are referring to the same number of active weights, but other notions of same capacity can also be used. We briefly discuss this in Appendix A.1.
  We compare the average absolute Kendall Rank correlation between different formulations of gradient flow and generalization. The subscript denotes the p-norm (l1 or l2 norm). We see that EGF has higher absolute correlation when compared to standard gradient flow measures. We also see that is consistent across Fashion MNIST (see Appendix B.1).
sections.heading: 2.2 MEASURING GRADIENT FLOW
sections.text: Gradient flow (GF) is used to study optimization dynamics and typically approximated by taking the norm of the gradients of the network (Pascanu et al., 2013; Nocedal et al., 2002; Chen et al., 2018; Wang et al., 2020; Evci et al., 2020).
  We consider a feedforward neural network f : RD → R, with function inputs x ∈ RD and network weights θ. The gradient norm is usually computed by concatenating all the gradients of a network into a single vector, g = ∂C∂θ , whereC is our cost function. Then the vector norm is taken as follows:
  gfp = ||g||p, (5)
  where p denotes the pth-norm.
  Effective Gradient Flow Traditional measures of gradient flow take the l1 or l2 norm of all the gradients (Chen et al., 2018; Pascanu et al., 2013; Evci et al., 2020). This is not appropriate for sparse networks, as this would include gradients of masked weights which have no influence on the forward pass. Furthermore, computing l1 or l2 across all weights in the networks gives disproportionate influence to layers with more weights. We instead propose a simple modification of Equation 5, which we term Effective Gradient Flow (EGF), that computes the average, masked gradient (only gradients of active weights) norm across all layers.
  We calculate EGF as follows:
  g = ( ∂C ∂θ1 m1, ∂C ∂θ2 m2, . . . , ∂C ∂θC mL) for l = 1, . . . , L (6)
  egfp = ∑L n=1 ||gi||p L , (7)
  where L is number of layers and ∂C ∂θl ml denotes an element-wise product of the gradients of layer l, ∂C ∂θl
  , and the mask ml applied to the weights of layer l. For a fully dense network, ml is a matrix of all ones, since no gradients are masked.
  EGF has the following favourable properties:
  • Gradient flow is evenly distributed across layers EGF distributes the gradient norm across the layers equally, preventing layers with a lot of weights from dominating the measure and also preventing layers with vanishing gradients from being hidden in the formulation, as is the case with equation 5 (when all gradients are appended together).
  • Only gradients of active weights are used EGF ensures that for sparse networks, only gradients of active weights are used. Even though weights are masked, their gradients are not necessarily
  • Possibility for application in gradient-based pruning methods Tanaka et al. (2020) showed that gradient-based pruning methods like GRASP (Wang et al., 2020) and SNIP (Lee et al., 2018a), disproportionately prune large layers and are susceptible to layer-collapse, which is when an algorithm prunes all the weights in a specific layer. Due to the fact that EGF is evenly distributed across layers, maintaining EGF (as opposed to standard gradient norm) could possibly be used as a pruning criteria. Furthermore, current approaches measuring or approximating the change in gradient flow during pruning in sparse networks (Wang et al., 2020; Evci et al., 2020; Singh Lubana & Dick, 2020), could benefit from this new formulation.
  To evaluate EGF against other standard gradient norm measures, such as the l1 and l2 norm, we empirically compare these measures and their correlation to test loss and accuracy. We take the absolute average of the Kendall Rank correlation (Kendall, 1938), across the different experiment configurations. We follow a similiar approach to Jiang et al. (2019), but unlike their work which has focused on correlating network complexity measures to the generalization gap, we measure the correlation of gradient flow to performance (accuracy and loss). We measure gradient flow at 10 points evenly spaced throughout training, specifically at the end of epoch 0, 99. 199, 299, 399, 499, 599, 699, 799, 899 and 999.
  Our results from Table 1 shows that EGF has a higher average absolute correlation to both test loss and accuracy. This is also true of Fashion MNIST (see Appendix B.1). Due to the comparative benefits of EGF, we use it for the remainder of the paper to measure the impact of interventions. We include all measures of gradient flow in Appendix B for completeness.
sections.heading: 2.3 ARCHITECTURE, NORMALIZATION, REGULARIZATION AND OPTIMIZER VARIANTS
sections.text: We briefly describe our key experiment variants below, and also include for completeness all unique variants in Table 2.
  Activation functions ReLU networks (Nair & Hinton, 2010) are known to be more resilient to vanishing gradients than networks that use Sigmoid or Tanh activations, since they only result in vanishing gradients when the input is less than zero, while on active paths, due to ReLU’s linearity, the gradients flow uninhibited (Glorot et al., 2011). Although most experiments are run on ReLU networks, we also explore different activation functions, namely PReLU (He et al., 2015), ELU (Clevert et al., 2015), Swish (Ramachandran et al., 2017), SReLU (Jin et al., 2015) and Sigmoid (Neal, 1992).
  Batch normalization and Skip Connections Other methods to help alleviate the vanishing gradient problem include the addition of skip connections (every two layers) (Srivastava et al., 2015; He et al., 2016) and batch normalization (Ioffe & Szegedy, 2015). We empirically explore these methods.
  Optimization and Regularization techniques We explore the impact of popular regularization methods: weight decay/l2 regularization (0.0001) (Krogh & Hertz, 1992; Hanson & Pratt, 1989) and data augmentation (random crops and random horizontal flipping (Krizhevsky et al., 2012)). Furthermore, we benchmark the impact of the most widely used optimizers such as minibatch stochastic gradient descent (with momentum (0.9) (Sutskever et al., 2013; Polyak, 1964) and without momen-
  tum (Robbins & Monro, 1951)) , Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSProp (Hinton et al., 2012).
sections.heading: 3 EMPIRICAL SET-UP
sections.text: SC-SDC MLP Setting We use the SC-SDC empirical setting (section 2.1) for all experiment variants. We train over 6000 MLPs for 1000 epochs and evaluate performance on CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009). We compare sparse and dense networks across various widths, depths, learning rates, regularization and optimization methods as shown in Table 2.
  We choose a max network width NMaxW of n+ 4, where n is the input dimension of the network. In the case of CIFAR, n = 3072 and so our maximum width NMaxW = 3076. We repeat these experiments with one, two and four hidden layers, with the number of active weights in these networks ranging from 949, 256 to 31, 765, 568 weights. In section 4, we discuss results achieved using four hidden layers on CIFAR-100 and we provide the one and two hidden layer results in Appendix C.
  Dense Width Following from SC-SDC, these networks are compared at various network widths, specifically a width of 308, 923, 1538, 2153, 2768 (10%, 30%, 50%, 70% and 90% of our maximum width NMaxW (3076)) as shown in Table 2. We use the term dense width to refer to the width of a network if that network was dense. For example, when comparing sparse and dense networks at a dense width of 308, this means the dense network has a width of 308, while the sparse network has a width of NMaxW (3076), but has the same number of active connections as the dense counterpart. We provide more detailed discussion of the choices made in the SC-SDC implementation in Appendix A.1.
  Extended CNN Setting We also extend our experiments to Wide Resnet-50 (the WRN-28-10 variant) (Zagoruyko & Komodakis, 2016) and use the optimization and regularization configurations from the paper.
sections.heading: 4 RESULTS AND DISCUSSION
sections.text: 4.1 COMPARISON OF DENSE AND SPARSE INTERVENTIONS USING SC-SDC
  In this section, we use the results of the Wilcoxon signed rank test from SC-SDC to identify where optimization choices are currently well suited for sparse networks and which are not. Furthermore, for each variant, we also measure the gradient flow using EGF(egf2 (6))1 as described in the previous section. Our main findings show that:
  1. Batch normalization is critical to training sparse networks, more so than it is for dense networks. This suggests that gradient instability is a key obstacle for sparse optimization.
  2. Weight decay (with and without batch normalization) and data augmentation ( without batch normalization) can hurt both sparse and dense network optimization. This particularly occurs when using accelerated, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam and RMSProp (Ruder, 2016). In these methods, large EGF (gradient flow) strongly correlates to poor test accuracy.
  3. Non-saturating activation functions, such as Swish (Ramachandran et al., 2017) and PReLU (He et al., 2015), achieve promising results in both the sparse and dense regime. However, these results are more statistically significant when sparse networks are used and so this could motivate for the use of similar activation functions for training sparse networks.
  Batch normalization plays a disproportionate role in stabilizing sparse networks Batch normalization ensures that the distribution of the nonlinearity inputs remains stable as the network trains, which was hypothesized to help stabilize gradient propagation (gradients do not explode or vanish) (Ioffe & Szegedy, 2015). Following from Table 3a and 3b, we see that batch normalization is statistically more important for sparse network performance than it is for dense networks, across most configurations and learning rates.
  1Note we measure EGF at 10 points throughout training and take the average
sections.heading: NR - No Regularization, BN - Batchnorm, SC - Skip Connections, DA - Data Augmention and L2- weight decay.
sections.text: From Table 3a and Figure 10a, we see methods such as L2 and data augmentation usually favour dense networks (apart from Adam and RMSProp, when using data augmentation). However, with the addition of batch normalization (Table 3b, L2 to L2 BN and DA to DA BN ), these methods favour sparse variants. This is especially apparent in Figure 10a, where batch normalization improves performance across all sparse optimizers, while resulting in a lower, more stable EGF. This further emphasizes the importance of stabilizing gradient flow, particularly in sparse networks.
  Weight Decay and Data Augmentation can hurt sparse network optimization When we take a closer look at the effects of weight decay and data augmentation on sparse network accuracy (Figure 2,Figure 13), we see that weight decay (even with batch normalization) drastically decreases accuracy when used with adaptive optimization methods that use an exponentially decaying average of past squared gradients (Adam and RMSProp). Furthermore, it results in distinctively larger EGF values, which hints at Adam and RMSProp being more sensitive to larger gradient norms than other optimizers. This agrees with Loshchilov & Hutter (2017), who proposed a different formulation of weight decay for adaptive methods, since the current L2 regularization formulation for adaptive methods could lead to weights with large gradients being regularized less, although this was not experimentally verified. In the context of data augmentation, we see poor test accuracy when it is used without batch normalization (Figure 10a). If used with batch normalization (Figure 2), it results in a lower EGF and best test accuracy. This further emphasized the need to stabilize gradient flow and how EGF can be used to this end.
  The potential of non-saturating activation functions - Swish and PReLU We also explore the effect of different activation functions on sparse network optimization. For the activation function
sections.heading: NR - No Regularization, BN - Batchnorm, SC - Skip Connections, DA - Data Augmention and L2- weight decay.
sections.text: variants, the best configuration for each optimizer was chosen. For Adagrad, Adam and RMSProp we use BN, SC and DA, while for SGD, we use BN, SC, L2 and DA.
  From Table 3c, we see that Swish, PReLU and Sigmoid favour sparse architectures, but from the performance results from Figure 12, we see that only Swish and PReLU are viable activation choices. We continue to see a consistent trend for adaptive methods (most notably in Adam and RMSProp), that higher EGF values, for example in SReLU, correspond to poor performance (Figure 11b), while promising methods result in a lower EGF value (such as Swish). This further emphasizes how EGF can be used to guide advances in network optimization.
sections.heading: 4.2 GENERALIZATION OF RESULTS ACROSS ARCHITECTURE TYPES.
sections.text: In this section, we move on from SC-SDC and extend our result to Wide ResNet-50. We note from Figure 3, that most of our results from SC-SDC also hold on larger, more complicated models. We see that L2 regularization (even with batch normalization) hurts performance for adaptive methods (Adagrad and Adam) and also results in higher EGF values (Figure 14). Furthermore, we also see data augmentation is beneficial when used with batch normalization. Finally, we see that Swish is a promising activation function for adaptive methods and leads to lower EGF (Figure 14). This shows that the SC-SDC results are not constrained to small scale experiments and that it can be used to learn about dynamics of larger, more complicated networks.
sections.heading: 5 RELATED WORK
sections.text: Pruning at Initialization Methods that prune at initialization aim to start sparse, instead of first pre-training an overparameterized network and then pruning. These methods use certain criteria to estimate at initialization, which weights should remain active. This criteria includes using the connection sensitivity (Lee et al., 2018b), gradient flow (via the Hessian vector product) (Wang et al., 2020) and conversation of synaptic saliency (Tanaka et al., 2020). Another branch of pruning is Dynamic Sparse Training, which uses information gathered during the training process, to dynamically update the sparsity pattern of these sparse networks (Mostafa & Wang, 2019; Bellec et al., 2017; Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019; Evci et al., 2019). While our work is motivated by the same goal of allowing networks to start sparse and converge to the same performance as dense networks, we instead focus on the impact of optimization and regularization choices on sparse networks.
  Sparse Network Optimization as Pruning Criteria Optimization in sparse networks has often been neglected in favour of studying network initialization. However, there has been work that has looked at sparse network optimization from different perspectives, mainly as a guide for pruning criteria. This includes using gradient information (Mozer & Smolensky, 1989; LeCun et al., 1989; Hassibi & Stork, 1992; Karnin, 1990), approximates of gradient flow (Wang et al., 2020; Dettmers & Zettlemoyer, 2019; Evci et al., 2020) and Neural Tangent Kernel (NTK) (Liu & Zenke, 2020) to guide the introduction of sparsity.
  Sparse Network Optimization to study Network Dynamics Apart from use as pruning criteria, optimization information has been used to investigate aspects of sparse networks, such as their loss landscape (Evci et al., 2019), how they are impacted by SGD noise (Frankle et al., 2019a), the effect of different activation functions (Dubowski, 2020) and their weight initialization (Lee et al., 2019). Our work differs from these approaches as we consider more aspects of the optimization and regularization process in a controlled experimental setting (SC-SDC), while using EGF to reason about some of the results.
sections.heading: 6 CONCLUSION AND FUTURE WORK
sections.text: In this work, we take a wider view of sparse optimization strategies and introduce appropriate tooling to measure the impact of architecture and optimization choices on sparse networks (EGF , SC-SDC ). Our results show that weight decay and data augmentation can hurt optimization, when adaptive optimization methods are used and this usually corresponds to a much higher EGF.Furthermore, we show how batch normalization is critical to training sparse networks, more so than it is for dense networks as it helps stabilize gradient flow. We also show the potential of non-saturating activation functions for sparse networks such as Swish and PReLU. Finally, we show that our results extend to more complicated models like Wide ResNet-50.
sections.heading: A SC-SDC
sections.text: In this section, we provide more information about SC-SDC and its benefits.
  A.1 SC-SDC IMPLEMENTATION DETAILS
  Wilcoxon Signed Rank Test This is a non-parametric test that compares dependent or paired samples, without assuming the differences in between the paired experiments are normally distributed (McDonald, 2009; Demšar, 2006).
  Random Sparsity Our work focuses on the training dynamics of random, sparse networks. This ensures that what is learned is not dependent on a specific pruning method, but rather can be used to better understand sparse training in general. Going forward, it would be interesting to explore these dynamics on pruned networks.
  We achieve random sparsity, by generating a random mask for each layer and then multiply the weights by this mask during each forward pass. The sparsity is distributed evenly across the network. For example, a 20% sparse MLP has 20% of the weights remaining in each layer.
  Dense Width A critical component to how we specify our experiments is a term we define as dense width. In order to fairly compare sparse and dense networks, we need them to have the same number of active connections at each depth. In the case of sparse networks, this means ensuring they have the same number of active connections as the dense networks, while remaining sparse. Dense width refers to the width of a network if that network was dense. This process of comparing sparse and dense networks at different dense widths is illustrated in figure 5.
  Fair comparison of Sparse and Dense networks As can be seen from figure 5, SC-SDC ensures the exact same active parameter count, but the sparse networks will be connected to more neurons. It is possible that the increased number of activations being used can lead to sparse networks having higher representational power, however most work on expressivity of neural networks looks at this from a depth perspective and proves certain depths of networks are universal approximators (Eldan & Shamir, 2016; Hornik et al., 1989; Funahashi, 1989).
  To this end, we ensure these networks have the same depth, but we believe going forward an interesting direction would be ensuring they have a similar amount of active neurons.
  SC-SDC comparison details For completeness, we provide more details of how we ensure sparse and dense networks are of the same capacity.
  Following from equation 2, to ensure the same number of weights in sparse and dense networks, we can ensure they have the same number of active weights at each layer as follows:
  ||alS ||0 = ||alD||0, for l = 1, . . . , L (8)
  This is achieved by masking each of the weight layers of sparse network S:
  alS = θ l S ml for l = 1, . . . , L (9)
  , where ml is a random binary matrix (mask) for layer l, s.t. ||ml||0 = alD, where alD is determined by the chosen capacity, these networks will be compared at.
  For SC-SDC, we need a maximum network width NMaxW and comparison width NW . We choose a max network width NMaxW of n+ 4, where n is the input dimension of the network. In the case of CIFAR, n = 3072 and so our maximum width NMaxW = 3076. The choice of n + 4 follows from Lu et al. (2017), where the authors prove a universal approximation theorem for width-bounded ReLU networks, with width bounded to n + 4. Our comparison width, NW , is equivalent to dense widths we vary in our experiments - 308, 923, 1538, 2153, 2768.
  The dimensions of each of layers are as follows:
  1. First Layer:
  θ1D ∈ RI×NW , θ1S ∈ RI×NMaxW , m1 ∈ {0, 1}I×NMaxW (10)
  2. Intermediate Layers:
  θ1S ∈ RNW×NW , θ1S ∈ RNMaxW×NMaxW , m{2,...,L−1} ∈ {0, 1}NMaxW×NMaxW (11)
  3. Final Layer:
  θLS ∈ RNW×O , θLS ∈ RNMaxW×O , mL ∈ {0, 1}NMaxW×O (12)
  , where NMaxW is maximum width of the sparse layer, NW is the comparison width, I is the input dimension, O is output dimension, L is the number of layers in the network, θlS is the weights in layer l of sparse network S and θlD is the weights in layer l of dense network D.
  This process would be the same for convolutional layers, but there would be a third dimension to handle the different channels. In figure 4, we provide an illustrative example showing how to ensure sparse and dense networks are compared fairly.
  A.2 BENEFITS
  The benefits of SC-SDC can be summarized as follows:
  • We can better understand sparse network optimization. SC-SDC allows us to identify which optimization or regularization methods are poorly suited to sparse networks in a controlled setting, ensuring the results are a direct result of the sparse connections themselves.
  • Learn at what parameter and size budget, sparse networks are better than dense. Comparing sparse and dense networks of the same capacity allows us to see which architecture is better at different configurations. In configurations where sparse architectures perform better, we could exploit advances in sparse matrix computation and storage (Zhao et al., 2018; Merrill & Garland, 2016) to simply default to sparse architectures.
sections.heading: B GRADIENT FLOW
sections.text: B.1 EGF RESULTS ON FMNIST
  We extend our experiments to Fashion MNIST (Xiao et al., 2017), a dataset that is distinctively different to the CIFAR datasets we used in section 2.2. We ran 450 experiments with networks with four hidden layers, using 0.001 as a learning rate and for 500 epochs. We varied configurations as follows:
  • Optimizers - Adagrad, Adam and SGD with momentum. • Regularization methods - no regularization, batchnorm, skip connections, l2 (0.0001) and
  data augmentation.
  From table 4, we see that out of the gradient flow formulations, EGF still correlates better to generalization performance.
  For completeness, we present the full set of results using the different formulations of gradient flow on CIFAR-100. Namely, we show ||g||1 (5) (Figure 6) ,||g||2 (5) (Figure 7) ,egf1 (6) (Figure 8) and egf2 (6) (Figure 9).
sections.heading: C DETAILED RESULTS FOR SC-SDC
sections.text: In this section, we presented the detailed results for our experiments.
  1. Detailed Results with a low learning rate (0.001). 2. Detailed Results with a high learning rate (0.1). 3. Results for different activation functions.
  Adagrad 53.933 +/- 3.137 53.837 +/- 2.903 58.118 +/- 2.059 56.836 +/- 5.211 53.755 +/- 3.108 44.738 +/- 18.207 55.204 +/- 3.483 57.153 +/- 1.194 53.896 +/- 3.159 55.491 +/- 1.362 Adam 24.538 +/- 17.34 36.206 +/- 22.015 46.911 +/- 5.617 49.115 +/- 6.169 50.875 +/- 0.63 43.025 +/- 17.121 58.379 +/- 1.538 58.947 +/- 1.261 52.981 +/- 0.918 54.54 +/- 0.895 RMSProp 24.241 +/- 15.653 45.916 +/- 12.861 53.122 +/- 3.217 54.358 +/- 3.115 50.249 +/- 0.631 41.931 +/- 17.325 58.101 +/- 1.318 58.665 +/- 1.248 52.356 +/- 1.692 53.987 +/- 0.756 SGD 50.469 +/- 1.471 36.697 +/- 16.941 60.295 +/- 1.784 46.193 +/- 18.397 50.389 +/- 1.562 36.478 +/- 16.985 52.78 +/- 2.313 55.299 +/- 0.347 51.949 +/- 1.698 53.941 +/- 1.082 SGD with mom(0.9) 54.749 +/- 1.473 45.291 +/- 14.629 61.43 +/- 2.792 53.408 +/- 19.158 54.382 +/- 1.506 44.661 +/- 14.911 56.496 +/- 2.326 58.336 +/- 0.632 54.298 +/- 1.738 53.312 +/- 3.529
  Adagrad 5.158 +/- 0.637 4.361 +/- 1.604 2.021 +/- 0.752 1.838 +/- 0.697 3.819 +/- 1.308 3.172 +/- 0.743 2.802 +/- 0.29 2.396 +/- 0.225 4.897 +/- 0.682 3.625 +/- 1.131 Adam 11.182 +/- 11.38 95.498 +/- 157.421 2.722 +/- 0.382 2.816 +/- 0.681 3.608 +/- 0.171 3.51 +/- 0.671 7.204 +/- 0.9 7.181 +/- 1.162 244.704 +/- 90.922 247.345 +/- 134.383 RMSProp 12.352 +/- 9.63 191.175 +/- 318.66 5.452 +/- 2.659 6.504 +/- 4.251 3.647 +/- 0.322 3.718 +/- 0.857 7.645 +/- 0.867 7.623 +/- 1.159 194.814 +/- 77.03 199.102 +/- 106.311 SGD 5.139 +/- 0.855 3.578 +/- 1.253 1.173 +/- 0.033 1.518 +/- 0.472 4.812 +/- 0.829 3.458 +/- 1.102 2.779 +/- 0.365 2.243 +/- 0.132 4.311 +/- 0.815 2.442 +/- 0.788 SGD with mom(0.9) 5.206 +/- 0.935 4.974 +/- 1.657 3.395 +/- 0.357 2.471 +/- 0.69 3.496 +/- 0.636 3.499 +/- 0.903 2.927 +/- 0.569 2.453 +/- 0.057 4.931 +/- 0.974 4.506 +/- 1.304
  Adagrad 24.956 +/- 1.686 22.189 +/- 2.633 25.479 +/- 1.175 22.624 +/- 3.991 24.556 +/- 1.343 16.263 +/- 8.572 27.008 +/- 4.51 29.011 +/- 2.027 26.852 +/- 1.781 27.279 +/- 1.334 Adam 15.583 +/- 3.736 22.713 +/- 3.209 8.865 +/- 3.933 10.306 +/- 6.494 19.646 +/- 0.889 17.555 +/- 8.751 29.652 +/- 1.801 30.013 +/- 1.688 23.748 +/- 1.504 25.151 +/- 0.657 RMSProp 4.97 +/- 8.083 17.983 +/- 9.803 11.597 +/- 7.201 20.396 +/- 3.643 19.835 +/- 0.637 16.678 +/- 9.06 29.501 +/- 1.853 29.667 +/- 1.86 21.43 +/- 7.297 21.363 +/- 9.074 SGD 20.809 +/- 0.911 11.776 +/- 8.53 26.802 +/- 1.492 14.004 +/- 10.133 20.695 +/- 0.743 11.375 +/- 8.282 24.511 +/- 3.541 27.515 +/- 1.428 23.805 +/- 1.551 22.871 +/- 7.053 SGD with mom(0.9) 25.269 +/- 1.456 17.466 +/- 8.93 29.958 +/- 3.146 24.143 +/- 12.147 24.741 +/- 1.38 16.867 +/- 8.621 27.301 +/- 3.007 28.893 +/- 1.526 26.817 +/- 1.83 25.595 +/- 3.516
  Adagrad 9.714 +/- 3.113 10.018 +/- 5.165 4.584 +/- 1.222 4.181 +/- 1.036 9.112 +/- 2.96 10.83 +/- 4.376 4.275 +/- 0.491 3.784 +/- 0.042 7.227 +/- 2.173 6.592 +/- 2.377 Adam 209.854 +/- 400.362 10867.689 +/- 10783.68 5.044 +/- 0.288 8.551 +/- 3.795 32.786 +/- 12.029 15.217 +/- 12.419 19.798 +/- 6.003 20.605 +/- 6.057 779.235 +/- 319.831 715.685 +/- 395.333 RMSProp 222.053 +/- 450.311 3680.784 +/- 5265.27 14.482 +/- 16.75 5.609 +/- 0.51 22.649 +/- 5.115 15.09 +/- 9.165 20.313 +/- 5.053 21.787 +/- 3.96 649.551 +/- 366.815 443.976 +/- 348.581 SGD 10.394 +/- 1.219 5.936 +/- 2.56 3.082 +/- 0.064 3.786 +/- 0.619 10.222 +/- 1.053 5.961 +/- 2.521 4.286 +/- 0.652 3.704 +/- 0.128 8.212 +/- 0.61 4.608 +/- 1.706 SGD with mom(0.9) 13.425 +/- 2.123 12.899 +/- 4.883 7.215 +/- 0.572 5.239 +/- 1.298 9.114 +/- 1.778 9.882 +/- 3.843 4.724 +/- 0.861 4.135 +/- 0.203 10.16 +/- 1.981 8.783 +/- 0.955
  Adagrad 19.469 +/- 10.563 31.377 +/- 15.801 22.62 +/- 16.083 37.885 +/- 17.664 19.739 +/- 12.699 40.133 +/- 14.52 56.194 +/- 1.319 57.536 +/- 0.712 15.615 +/- 9.853 30.756 +/- 17.113 Adam 10.0 +/- 0.0 10.0 +/- 0.005 10.0 +/- 0.0 10.0 +/- 0.0 9.998 +/- 0.012 9.98 +/- 0.036 53.191 +/- 2.687 54.964 +/- 1.88 9.999 +/- 0.003 10.001 +/- 0.005 RMSProp 9.999 +/- 0.003 10.001 +/- 0.01 10.0 +/- 0.0 10.001 +/- 0.003 10.0 +/- 0.0 10.191 +/- 0.578 53.328 +/- 1.022 53.62 +/- 1.056 10.0 +/- 0.004 10.001 +/- 0.011 SGD 57.603 +/- 1.546 56.487 +/- 3.866 51.81 +/- 6.396 60.065 +/- 1.465 8.255 +/- 19.659 30.631 +/- 26.518 59.191 +/- 1.805 60.295 +/- 0.949 57.599 +/- 1.302 56.398 +/- 2.893 SGD with mom(0.9) 9.303 +/- 2.608 12.872 +/- 14.06 10.0 +/- 0.0 19.086 +/- 18.81 10.579 +/- 10.646 35.2 +/- 23.74 57.822 +/- 1.012 59.058 +/- 0.886 9.334 +/- 2.582 9.117 +/- 9.037
  Adagrad 4.332 +/- 3.705 6.299 +/- 3.469 2.036 +/- 0.344 1.895 +/- 0.263 3.21 +/- 1.888 3.186 +/- 1.288 5.979 +/- 0.403 5.19 +/- 1.109 3.934 +/- 2.999 5.788 +/- 3.694 Adam 2.328 +/- 0.066 6.323 +/- 11.003 2.303 +/- 0.0 2.303 +/- 0.0 405749.405 +/- 1051430.43 343515.597 +/- 1330067.381 51.96 +/- 34.783 60.026 +/- 18.807 4.174 +/- 7.207 2.366 +/- 0.17 RMSProp 1339.999 +/- 5147.876 275.11 +/- 831.374 2.303 +/- 0.0 2.303 +/- 0.002 212.06 +/- 812.388 43851.25 +/- 122539.555 40.846 +/- 17.869 107.835 +/- 42.366 30.677 +/- 104.926 349.709 +/- 1315.569 SGD 6.609 +/- 1.155 7.01 +/- 2.525 4.964 +/- 1.373 3.744 +/- 1.348 0.339 +/- 0.897 1.659 +/- 1.237 3.758 +/- 0.389 3.254 +/- 0.341 4.634 +/- 0.791 4.674 +/- 1.132 SGD with mom(0.9) 2.138 +/- 0.615 3.166 +/- 4.273 2.303 +/- 0.0 2.129 +/- 0.359 1.797 +/- 0.975 2.175 +/- 0.895 6.208 +/- 0.244 5.396 +/- 0.905 2.149 +/- 0.595 4.388 +/- 8.928
  Adagrad 4.832 +/- 3.339 11.731 +/- 7.078 10.304 +/- 7.42 14.827 +/- 8.334 5.383 +/- 5.543 14.899 +/- 9.16 28.804 +/- 1.905 29.826 +/- 1.857 5.885 +/- 3.829 11.649 +/- 6.443 Adam 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 0.999 +/- 0.007 22.995 +/- 1.308 23.792 +/- 2.673 1.0 +/- 0.0 1.001 +/- 0.003 RMSProp 1.0 +/- 0.0 1.001 +/- 0.004 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.0 1.0 +/- 0.008 24.424 +/- 1.281 25.13 +/- 1.514 0.999 +/- 0.003 0.999 +/- 0.005 SGD 25.719 +/- 11.926 23.833 +/- 11.53 11.46 +/- 5.91 22.79 +/- 10.489 28.546 +/- 8.056 24.077 +/- 11.917 30.83 +/- 2.898 31.749 +/- 1.631 26.638 +/- 9.891 28.838 +/- 3.675 SGD with mom(0.9) 1.0 +/- 0.0 5.53 +/- 6.788 1.0 +/- 0.0 8.005 +/- 9.069 4.471 +/- 5.342 12.283 +/- 10.29 31.041 +/- 2.264 31.778 +/- 1.709 1.003 +/- 0.009 1.689 +/- 2.477
  Adagrad 7.667 +/- 4.947 14.836 +/- 8.468 4.13 +/- 0.358 5.685 +/- 2.939 6.024 +/- 3.425 6.571 +/- 2.298 9.474 +/- 0.926 8.717 +/- 1.854 9.247 +/- 5.927 15.916 +/- 9.915 Adam 4.664 +/- 0.201 8.052 +/- 8.361 4.605 +/- 0.0 4.605 +/- 0.0 27.215 +/- 87.569 64.341 +/- 137.011 170.973 +/- 84.218 367.302 +/- 141.566 4.631 +/- 0.081 4.752 +/- 0.508 RMSProp 11.792 +/- 23.242 57.154 +/- 202.471 4.605 +/- 0.0 4.606 +/- 0.003 4.605 +/- 0.0 5404.823 +/- 14437.166 146.552 +/- 73.377 748.996 +/- 361.425 5734.776 +/- 20650.808 5.919 +/- 3.307 SGD 10.48 +/- 2.908 11.058 +/- 3.433 9.797 +/- 5.488 11.97 +/- 5.69 5.7 +/- 1.647 5.322 +/- 0.59 5.587 +/- 0.934 5.004 +/- 0.265 11.017 +/- 7.566 8.611 +/- 1.653 SGD with mom(0.9) 4.605 +/- 0.0 12.49 +/- 11.831 4.605 +/- 0.0 4.367 +/- 0.342 3.903 +/- 1.591 4.965 +/- 0.518 10.612 +/- 1.41 9.014 +/- 2.289 4.607 +/- 0.005 16.577 +/- 29.665