We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments with CIFAR-10, along with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.
Summary
This paper studied the generalization power of CNNs and showed several upper bounds of generalization errors. Their results have two characteristics. First, the bounds are in terms of the quantity that is independent of the input dimension (size-free). Second, the upper bounds involve the distance between initial and learned parameters. These results improved the upper bounds that we can derive by naively applying the results of Bartlett et al. (2017) or Neushubar et al. (2017), because the dominant term of the existing upper bounds contained $l_{2, 1}$ or $l_2$ norms, which could depend on the input dimensions in the worst case. The authors empirically showed that there is a correlation between the generalization error of learned CNNs and the dominant term of the upper bound (i.e., the product of the parameter size and the distance from the set of initial parameters).
Decision
To the best of my knowledge, this is the first work that proved the size-free generalization bound for multi-layer CNNs. However, I think the assumption on the hypothesis class is very restrictive and significantly eases the problem, as I discuss in detail later. Therefore, I judge the technical contribution of the paper is moderate and recommend to reject the paper weakly.
By the standard argument of the statistical learning theory (such as Theorem A.4), we can typically bound the generalization error by $O(B\sqrt{D/N})$ where $B$ is the infimum of Lipschitz constant of hypotheses, $D$ is the intrinsic dimension of the hypothesis class, and $N$ is the sample size. Therefore, we can derive the size-free generalization bound if $B$ does not depend on the input dimension. Since the hypothesis class $F_\beta$ is defined via the spectral norm of CNNs, it is not surprising that we can derive the size-freeness of $B$. The size-free generalization bound has been already proven by Du et al., (2017), although it was the two-layered case. They imposed a restricted eigenvalue assumption. I think it implies that we need more sophisticated analysis if we do not assume the size-freeness of the hypothesis class.
Comments
- The authors claimed that Figure 3 is consistent with theorems because, according to the upper bound of theorems, the distance from the initialization point decreases when the generalization error is the same and the parameter size increases. However, I think it is too aggressive to conclude it from Figure 3 because the decreasing trend in the value of $\|K-K_0\|_\sigma$ is found only around $2\times 10^6\leq W \leq 3\times 10^6$. Furthermore, the value of $\|K-K_0\|_\sigma$ for  $W\approx 5\times 10^5$ is approximately the same as the value for $W\approx 3\times 10^6$.
Suggestions
- Please add the conclusion section which summarizes the paper and discusses the possible research directions.
Minor Comments
- page 1, section 1, paragraph 1
	- ... with roots in (Bartett, 1998) , is that ... → Use \citet
- page 2, section 2.1, paragraph 2
	- Write the definition of "expansive" activations.
- page 3, section 2., theorem 2.1
	- I think we should replace $\log(\lambda n)$ and $\log(\lambda)$ in equations with $\log(\beta \lambda n)$ and $\log(\beta \lambda)$, respectively.
- page 3, section 2.2, definition 2.2
	- $N$ → $\mathbb{N}$