How Many Samples are Needed to Estimate a Convolutional Neural Network?
A widespread folklore for explaining the success of Convolutional Neural Networks (CNNs) is that CNNs use a more compact representation than the Fullyconnected Neural Network (FNN) and thus require fewer training samples to accurately estimate their parameters. We initiate the study of rigorously characterizing the sample complexity of estimating CNNs. We show that for an m-dimensional convolutional filter with linear activation acting on a d-dimensional input, the sample complexity of achieving population prediction error of ✏ is r Opm{✏ 2 q 2 , whereas the sample-complexity for its FNN counterpart is lower bounded by ⌦pd{✏ 2 q samples. Since, in typical settings m ! d, this result demonstrates the advantage of using a CNN. We further consider the sample complexity of estimating a onehidden-layer CNN with linear activation where both the m-dimensional convolutional filter and the r-dimensional output weights are unknown. For this model, we show that the sample complexity is r O`pm`rq{✏ 2˘w hen the ratio between the stride size and the filter size is a constant. For both models, we also present lower bounds showing our sample complexities are tight up to logarithmic factors. Our main tools for deriving these results are a localized empirical process analysis and a new lemma characterizing the convolutional structure. We believe that these tools may inspire further developments in understanding CNNs.
The authors consider the number of samples needed to achieve an error epsilon in the context  of learning an m-dimensional convolutional filter as well as one followed by a linear projection. This is motivated by a desire to rigorously understand the empirical success of CNNs.  This paper seems technically correct, yet I believe the setting is very far from real CNNs to the point where it’s not clear if the results will be impactful. The authors only consider a linear convolution layer, which corresponds to a wiener filtering-like operation according to their model, for removing noise for estimating the label. My concern is the motivation, the novelty and the assumptions.  For the assumptions, I am not sure that the non degeneracy assumption (A3) is that easy to constraint. Just  whitening looks complex, for example in the case of image: either one could use a basis a priori such as a wavelet basis but then this operation is quite complex wrt the tools used in this paper; a second option would be to whiten directly the signal that might be approximative somehow for very large signals  For novelty, the authors relate their work to relevant and very recent work on understanding and characterizing deep learning, however I am surprised there is no discussions of any classic signal processing/adaptive filters, the linear restriction in the model and the squared loss gives me a sense that there is a literature from signal processing not considered. In general the lack of literature before 2015 seems odd. I'd like the authors to relate their work to classic signal processing which considers in detail learning linear filters. Additionally I would suggest a relationship to the classic results such as (Barron, 1994) on one-hidden-layer networks.  Regarding the motivation, while I agree the results is interesting, since as the authors admit the result is quite distant from the non-linear activation case and  it is unclear to me if this result will be useful.