FROM HARD TO SOFT: UNDERSTANDING DEEP NETWORK NONLINEARITIES VIA VECTOR QUANTIZATION AND STATISTICAL INFERENCE
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling. In particular, DN layers constructed from these operations can be interpreted as max-affine spline operators (MASOs) that have an elegant link to vector quantization (VQ) and K-means. While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax. This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs). We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural "hard" VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding "soft" VQ inference problems. We further extend the framework by hybridizing the hard and soft VQ optimizations to create a β-VQ inference that interpolates between hard, soft, and linear VQ inference. A prime example of a β-VQ DN nonlinearity is the swish nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation. Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters.
This work extends the applicability of the spline theory of deep networks explored in previous works of Balestriero/ Baraniuk. The previous works setup DNs as layer-wise max-affine spline operators (MASOs) and recovers several non-linearities practically used as special cases of these MASOs. The previous works already recover RELU variants and some downsampling operators that the current submission characterizes as "hard" quantization.
The major contribution of this work is extending the application to "soft" quantization that recovers several new non-linear activations such as soft-max. It is well-known that the k-means algorithm can be considered as a run of an EM algorithm to recover the mean parameters of a gaussian mixture model. The "hard" to "soft" transformation, and any interpolation in between follows from combining this insight with the previous works. As such there isnt a major technical contribution imho in this work. Furthermore, the presented orthogonalization for easier inference has been used before in many works, some of which this submission also cites, most importantly in the previous work of Balestriero/ Baraniuk that this submission extends. 
Nevertheless there is value in novel results that may follow from previous works in a straightforward but non-trivial fashion, as long as it is well-presented and thoroughly researched and implication well-highlighted. This paper does that adequately, so I will suggest weak accept. Furthermore, this work could spark interesting future works and fruitful discussions at the ICLR. It is well-written and the experimental evaluation is adequate.
I would suggest a couple of ways to possibly improve the exposition. The paper is somewhat notation heavy. When considering single layers, the superscript for the layer could be dropped in favor of clarity. I would suggest moving the definition of MASOs to the main text, and present Proposition 8 in some form in the main text as well. To a reader not familiar with previous works, or with splines, this could be helpful. Use of orthogonalization could be highlighted not just a tool for tractability but also regularization. For inference on GMMs, it corresponds to a type of variational inference, which could be mentioned.