We analyze the expressiveness and loss surface of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers. We show that such CNNs produce linearly independent features at a "wide" layer which has more neurons than the number of training samples. This condition holds e.g. for the VGG network. Furthermore, we provide for such wide CNNs necessary and sufficient conditions for global minima with zero training error. For the case where the wide layer is followed by a fully connected layer we show that almost every critical point of the empirical loss is a global minimum with zero training error. Our analysis suggests that both depth and width are very important in deep learning. While depth brings more representational power and allows the network to learn high level features, width smoothes the optimization landscape of the loss function in the sense that a sufficiently wide network has a well-behaved loss surface with almost no bad local minima.
This paper presents an analysis of convolutional neural networks from the perspective of how the rank of the features is affected by the kinds of layers found in the most popular networks. Their analysis leads to the formulation of a certain theorem about the global minima with respect to parameters in the latter portion of the network.
The authors ask important questions, but I am not sure that they obtain important answers. On the plus side, I'm glad that people are trying to further our understanding our neural networks, and I think that their investigation is worthy of being published.
They present a collection of assumptions, lemmas, and theorems. They have no choice but to have assumptions, because they want to abstract away the "data" part of the analysis while still being able to use certain properties about the rank of the features at certain layers.
Most of my doubts about this paper come from the feeling that equivalent results could be obtained with a more elegant argument about perturbation theory, instead of something like the proof of Lemma A1. That being said, it's easy to voice such concerns, and I'm willing to believe that there might not exist a simple way to derive the same results with an approach more along the line of "whatever your data, pick whatever small epsilon, and you can always have the desired properties by perturbing your data by that small epsilon in a random direction". Have the authors tried this ?
I'm not sure if the authors were the first to present this approach of analyzing the effects of convolutions from a "patch perspective", but I think this is a clever approach. It simplifies the statement of some of their results. I also like the idea of factoring the argument along the concept of some critical "wide layer".
Good review of the literature.
I wished the paper was easier to read. Some of the concepts could have been illustrated to give the reader some way to visualize the intuitive notions. For example, maybe it would have been interesting to plot the rank of features a every layer for LeNet+MNIST ?
At the end of the day, if a friend asked me to summarize the paper, I would tell them :
"Features are basically full rank. Then they use a square loss and end up with an over-parametrized system, so they can achieve loss zero (i.e. global minimum) with a multitude of parameters values."
Nitpicking :
"This paper is one of the first ones, which studies CNNs."
This sentence is strange to read, but I can understand what the authors mean.
"This is true even if the bottom layers (from input to the wide layer) and chosen randomly with probability one."
There's a certain meaning to "with probability one" when it comes to measure theory. The authors are using it correctly in the rest of the paper, but in this sentence I think they simply mean that something holds if "all" the bottom layers have random features.