The effectiveness of deep neural architectures has been widely supported in terms of both experimental and foundational principles. There is also clear evidence that the activation function (e.g. the rectifier and the LSTM units) plays a crucial role in the complexity of learning. Based on this remark, this paper discusses an optimal selection of the neuron non-linearity in a functional framework that is inspired from classic regularization arguments. A representation theorem is given which indicates that the best activation function is a kernel expansion in the training set, that can be effectively approximated over an opportune set of points modeling 1-D clusters. The idea can be naturally extended to recurrent networks, where the expressiveness of kernel-based activation functions turns out to be a crucial ingredient to capture long-term dependencies. We give experimental evidence of this property by a set of challenging experiments, where we compare the results with neural architectures based on state of the art LSTM cells.
The scope of the paper is interesting: to additionally learn the nonlinear activation function of the neuron.
The insights provided in section 2 with eqs (2)-(5) are interesting and naturally build on the previous work of Poggio & Girosi (1990) and Smola (1998). I found this a nice new insight and the strongest part of the paper. It is e.g. revealing to see to which P and L the rectifier nonlinearity is corresponding.
On the other hand I also have a number of suggestions for further improvement:
- Section 1: related to the overall function to be learned, the authors state "this general problem has been already solved". I think this statement is not completely correct, because depending on the choice of the stabilizer one obtains different optimal representations (e.g. Gaussian RBF or thin plate splines) as explained in Poggio & Girosi (1990). The theory does not tell what the best stabilizer is.
Additional relevant work that would be good to mention at this point, in the area of kernel methods, is e.g. learning the kernel.
- It seems that no other existing work on deep kernel machines has been mentioned in the paper, while in the conclusions the authors state "In this paper we have introduced Kernel-based deep neural networks". 
- Related to the training set T_N the notation e^kappa is not explained. It is not clear how this is related to eq (1).
- It would be good to comment on the difference between (3)(4) and Poggio & Girosi (1990).
- unnumbered eq after (5): are there multiple solutions to the problem (non-convex)? 
- The explanation of the recurrent network at the end of section 2 is too limited. Moreover, LSTM is not just a neuron nonlinearity, but a recurrent network with a particular structure. To which P and L would LSTM correspond?
- Fig.2: some of the nonlinearities look quite complicated and some of them are oscillatory (is this desirable? it reminds us of overfitting). Often one is interested in activation functions with a "simple shape" like sigmoid, tanh, relu. A more complicated nonlinearity may reduce the interpretability of the model.
- The examples given are rather conceptual (though nice) examples of the proposed method. However, no comparisons with other methods have been made yet in terms of generalization performance, e.g. on a few standard classification benchmark data sets, in comparison with other deep or shallow models.
A possible drawback of the proposed method might be (or maybe not) that additional unknown parameters need to be learned, which could possibly lead to worse generalization. It might be good to further investigate this.