Self-Normalizing Neural Networks
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization schemes, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs, and other machine learning methods such as random forests and support vector machines. For FNNs we considered (i) ReLU networks without normalization, (ii) batch normalization, (iii) layer normalization, (iv) weight normalization, (v) highway networks, and (vi) residual networks. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep.
The paper proposes a new (class of) activation function f to more efficiently train very deep feed-forward neural networks (networks with f are called SNNs). f is a combination of scaled ReLU (positive x) and an exponential (negative x). The authors argue that 1) SNNs converge towards normalized activation distributions 2) SGD is more stable as the SNN approx preserves variance from layer to layer. In fact, f is part of a family of activation functions, for which theoretical guarantees for fixed-point convergence exist. These functions are contraction mappings and characterized by mean/variance preservation across layers at the fixed point -- solving these constraints allows finding other "self-normalizing" f, in principle.
        Whether f converges to a fixed point, is sensitive to the choice of hyper-parameters: the authors demonstrate certain weight initializations and parameters settings that give the fixed-point behavior. Also, authors analyze the performance of SNNs with a modified version of dropout, which preserves mean/variance while using f.
        In support, authors provide theoretical proofs and experimental results on many datasets, although these vary in size (and quality?).
        As such, I think the paper warrants an accept -- although I have questions / concerns that I hope the authors can address.
        Questions / concerns:
        1. Most exps are run on small datasets. This is understandable since only FNNs are considered, but do we get similar results on networks that also include e.g. convolutional / recurrent layers? How can the theoretical analysis work / apply in that case (e.g. convolutions / pooling / memory cells are not always contraction mappings)?
        2. Ad point 1. What are the results for SNNs on MNIST / Cifar-10?
        3. The authors only considered layers without bias units. What happens when these are included? (This likely make it harder to preserve domains, for instance)
        4. During training, the mean/var of the weights do change, and are not guaranteed to be in a basin of attraction (see Thm 1) anymore. Why is there still convergence to a fixed-point, even though no e.g. explicit weight clipping / normalization is applied (to make Thm 1 apply)?
        5. How does this analysis interact with different optimizers? What is the difference between using momentum-based / Adagrad etc for SNNs?
        6. Hyper-parameters of f need to be set precisely, otherwise f is not domain-preserving / contraction mapping anymore. How sensitive is the model to this choice?
        7. It is unclear how the fixed-point analysis theoretically relates to the generalization performance of the found solution. Is there a theoretical connection?