Polynomial neural networks are essentially polynomial functions. These networks are shown to have nice theoretical properties by previous analysis, but they are hard to train when their polynomial orders are high. In this work, we devise a new type of activations and then create the Ladder Polynomial Neural Network (LPNN). The LPNN has a feedforward structure. It provides good control of its polynomial order because its order increases by 1 with each of its hidden layers. The new network can be treated with deep learning techniques such as batch normalization and dropout. As a result, it can be well trained with generic optimization algorithms regardless of its depth. In our empirical study, deep LPNN models achieve good performances in a series of regression and classification tasks.
This paper proposes Ladder Polynomial Neural Networks (LPNNs) that use a new type of activation primitive -- a product activation -- in a feed-forward architecture. Unlike other polynomial architectures that grow in the order exponentially with network depth, the proposed approach gives explicit control over the order and smoothness of the network output and enables training with standard techniques.	
				
The proposed architecture is closely related to a decomposition of a k’th order multivariate polynomial function
[T, x^{\otimes k}] = \lambda^\top (A x \odot A x \odot …  \odot A x)	=  \lambda^\top (A x)^{\odot k}
where T is a symmetric tensor of polynomial coefficients and [\cdot,\cdot] denotes contraction. This is a shallow (one layer architecture) and sometimes referred as a Waring decomposition.
In this paper, the authors propose a specific chain factorization of the polynomial (Eq 5 in the paper), where they write the factors recursively, that they name as a ladder polynomial neural network. 
h^\ell = (W_\ell h^{\ell-1} \odot V^{\ell} x)
The ladder architecture is very closely related to tensor trains (https://epubs.siam.org/doi/10.1137/090752286). I found it surprising and somewhat alarming that this literature is not being cited as these methods are also quite well known in deep learning.
I like the smoothness analysis of section 3.1 -- the proof is quite easy to follow and direct. I would be quite surprised if this result would not be known in the literature in some other form but I don’t recall seeing it. On the other hand it seems to be inevitably very loose for a deep ladder network unless the network models the zero function. It would have been a valuable addition to the experimental section, if this bound would have been illustrated numerically on synthetic examples.
In 3.2, The authors say that the objective is multiconvex -- I would argue that it is multilinear (apart from the regularization term, that is later introduces). The observation in 3.3, that batch-normalization or dropout can be used for this model is perhaps tangential to the main argument. These is investigated in the experimental section but I don’t see a clear conclusion. The section in 3.4 must include links to tensor decompositions beyond factorization machines.
		
Overall, I think the paper has some merit and could be interesting for some readers, despite the fact that the contribution is not very original and the treatment could be improved in many ways.