We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Particularly, we present an algorithm framework, Hierarchical Measure Group and Approximate System (HMGAS), nicknamed SSystem, of which NNs are special cases. In addition to many other results, the framework enables us to prove that 1) NNs implement renormalization group (RG) using information geometry, which points out that the large scale property to renormalize is dual Bregman divergence and completes the analog between NNs and RG; 2) and under a set of realistic boundedness and diversity conditions, for large size nonlinear deep NNs with a class of losses, including the hinge loss, all local minima are global minima with zero loss errors, using random matrix theory.
The paper provides a new framework "S-System" as a generalization of hierarchal models including neural networks. The paper shows an alternative way to derive the activation functions commonly used in practice in a principled way. It further shows that the landscape of the optimization problem of neural networks has nice properties in the setting where the number of input/hidden units tending to infinity and the neurons satisfy certain diversity conditions.
Overall, the paper presents super interesting ideas that can potentially lead to a deeper understanding of the fundamentals of deep learning. However, for a general reader it is a hard-to-follow paper. Without a full understanding of the various domains this paper presents ideas from, it is hard to verify and fully understand the claims. I believe the paper would be better appreciated by an audience of a mathematical journal. As an alternative, I would encourage the readers to split the paper and possibly simplify the content by using a running example (more concrete than the one of MLP used) to explain the implications as well as assumptions.
A clearer, more accessible presentation is necessary so that a non-expert can understand the paper's results. Thus, I vote to reject.