A MAX-AFFINE SPLINE PERSPECTIVE OF RECURRENT NEURAL NETWORKS
We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs). We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator. The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper. First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time. Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input. Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit 2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization. Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions. In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.
This paper builds upon recent work by Balestriero and Baraniuk (ICML 2018) that concern max-affine spline opertaor (MASO) interpretation of a substantial class of deep networks. In the new paper a special focus is put on Recurrent Neural Networks (RNNs), and it is highlighted based on theoretical considerations leveraging the MASO and numerical experiments that in the case of a piecewise affine and convex activation function, using noise in initial hidden state acts as regularization.  
Overall I was impressed by the volume of contributions presented throughout the paper and also I very muched like the light shed on important classes of models that turn out to be not as black box as they could seem. My enthouasiasm was somehow tempered when discovering that the MASO modelling here was in fact a special case of Balestriero and Baraniuk (ICML 2018), but it seems that despite this the specific contribution is well motivated and justified, especially regarding application results. Yet, the other thing that has annoyed me and is causing me to only moderately champion the paper so far is that I found the notation heavy, not always well introduced nor explained, and while I believe that the authors have a clear understanding of things, it appears to me that the the opening sections 1 and 2 lack notation and/or conceptual clarity, making the paper hard to accept without additional care. To take a few examples:
a) In equation (3), the exponent (\ell) in A and B is not discussed. On a different level, the term "S" is used here but doesn't seem to be employed much in next instances of MASOs...why? 
b) In equation (4), sure you can write a max as a sum with an approxiate indicator (modulo unicity I guess) but then what is called Q^{(\ell)} here becomes a function of A^{(\ell)}, B^{(\ell)}, z^{(\ell-1)}...?
c) In proposition 1, the notation A_sigma is not introduced. Of course, there is a notation table later but this would help (to preserve the flow and sometimes clarify things) to introduce notations upon first usage...
d) Still in prop 1, braket notation not so easy to grasp. What is A[z]z? 
e) Still in prop 1, recall that sigma is assumed piecewise-linear and convex? 
f) In th1, abusive to say that the layer "is" a mapping, isn't it?  
g) In Theorem 2, what is f? A generic term for a deterministic function? 
Also, below the Theorem, "affine" or "piecewise affine"? 
h) I found section 4 somehow disconnected and flow-breaking. Put in appendix and use space to better explain the rest? 
i) Section 5 is a strong and original bit, it seems. Should be put more to the fore in abstract/intro/conclusion?