Optimal Architectures in a Solvable Model of Deep Networks
Deep neural networks have received a considerable attention due to the success of their training for real world machine learning applications. They are also of great interest to the understanding of sensory processing in cortical sensory hierarchies. The purpose of this work is to advance our theoretical understanding of the computational benefits of these architectures. Using a simple model of clustered noisy inputs and a simple learning rule, we provide analytically derived recursion relations describing the propagation of the signals along the deep network. By analysis of these equations, and defining performance measures, we show that these model networks have optimal depths. We further explore the dependence of the optimal architecture on the system parameters.
This paper describes a particular instantiation of a deep neural network with solvable dynamics during feedforward propagation of an input. They show for such an architecture, an optimal depth (number of layers) can be computed and determine analytically situations where higher depth is beneficial.
While this work attempts to elucidate a theoretical understanding of deep networks, the model they analyze diverges substantially from models which have been successful. For example, the weights in their model are 'trained' analytically in a layer-wise manner (via pseudo-inverse rule). Furthermore, the authors presume the function of each layer is to preserve the fundamental clustering structure of the input while progressively removing noise. There is no citation or work provided showing this empirically occurs in deep networks.  Unlike [1] which they cite as related theoretical work on deep learning, the do not connect their findings back to the models which are currently successful. Therefore, it is unclear how to connect their theoretical findings with the current field of deep learning.  That aside, the analysis and results do have close connections to hebbian learning and more connections should be made with that literature. Otherwise, the current connection to modern deep networks made in this work is tenuous.  [1] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear 274 dynamics of learning in deep linear neural networks. arXiv.org, December 2013.