On the Number of Linear Regions of Deep Neural Networks
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models compute functions that react equally to complicated patterns of different inputs. The compositional structure of these functions enables them to re-use pieces of computation exponentially often in terms of the network's depth. This paper investigates the complexity of such compositional maps and contributes new theoretical results regarding the advantage of depth for neural networks with piecewise linear activation functions. In particular, our analysis is not specific to a single family of models, and as an example, we employ it for rectifier and maxout networks. We improve complexity bounds from pre-existing work and investigate the behavior of units in higher layers.
I. Introduction1) Limited number of papers have investigated deep networks from a theoretical perspective?? I disagree with this statement, many papers from Bengio's group do just this. 2) Define what you mean by popular networks, this is too vague (is it CNNs, RNNs, LSTMs, different non-linearities)3) Pascanu did the analysis for ReLU, pls justify the added benefit for doing this for piecewise linear functions? How novel is your work compared to Pascanu-13, as he did this for rectified units. You should be more clear about your novelties compared to this paper.4) Pls quantify what significant improvement means??II. Feedforward Neural Networksa) It woudl be helpful to have some pictures to go along with 2.2, as Pascanu-13 gives, or include this in supplemental material and point to itb) does the idea of input space foldings also depend on the non-linearity or no? this should be stated more clearly?c) So what is the bound on \epsilon to guarantee it stable to perturbation, pls clarify?d) There is very little connection between sections. It would be better if the flow between sub-sections was better.III. Deep Rectifier Networksa) it would be helpful if you had a visual description for 3.1 in the paper (or supplemental material)b) provide a deeper analysis of what the theorem doesIV. Deep Maxout Networksa) again, pictures for this section would be helpful as wellb) a better comparison and analysis for linear regions for rectified and maxout would help the readers understand why one could be preferred over another. This is a weakness of the paperV. Conclusionsa) How do the proofs in Theorem 4 and 8 help to show that deep models generalize better to unseen examples