Fantastic Generalization Measures and Where to Find Them
Generalization of deep networks has been intensely researched in recent years, resulting in a number of theoretical bounds and empirically motivated measures. However, most papers proposing such measures only study a small set of models, leaving open the question of whether these measures are truly useful in practice. We present the first large scale study of generalization bounds and measures in deep networks. We train over two thousand CIFAR-10 networks with systematic changes in important hyper-parameters. We attempt to uncover potential causal relationships between each measure and generalization, by using rank correlation coefficient and its modified forms. We analyze the results and show that some of the studied measures are very promising for further research.
The paper addresses the well-known generalization problem for deep networks: why do these models successfully generalize even though they contain more parameters than the size of the training set? Recent works on this problem have proposed various measures of model complexity (that is, different from the number of parameters) with the claim that they (at least partially) capture the *true* capacity of deep networks in order to answer this question. The authors report the results of a large-scale, systematic study of the empirical effect of these complexity measures on the observed generalization behavior of deep networks. The empirical study of generalization behavior is inherently tricky, and the authors make a non-trivial contribution just by designing an experiment that can convincingly address this question. The authors give careful attention to various potential obstacles related to experimental design and statistical validity. The results of the authors* experiments are highly valuable to the research community. I recommend to accept. ********************************************************************************* Technical comments for the authors: -Please rewrite or elaborate on the first two paragraphs of section 6 for clarity. -It seems to make little sense to consider norm-based complexity measures that are not margin-normalized (or at least normalized in some other way). Please include a margin-normalized version of each such measure in Table 5; certain key quantities (e.g. *frob-distance*) have no corresponding margin-normalized entry. -The quantity *sum-of-frob/margin* is certainly a quantity of interest and it should be included in table 2 and possibly merits some discussion. -I disagree somewhat with the discussion of the spectral bound in section 7. In particular, I would not agree that the poor performance of *spec-orig* is surprising, because it contains an astronomically-sized *proof artifact* (as you call it). Namely, in this bound and in similar norm-based bounds, the term which is the product of the matrices in the network is (heuristically) unnecessary; in particular it (heuristically) ought to be able to be replaced with the network*s *average-case Lipschitz constant* (this is one of the issues which Arora et al. 2018 attempted to address; unfortunately formalizing an adequate notion of *average-case Lipschitz constant* can be quite cumbersome). This superfluous product of matrix norms is of course highly correlated with model size, so it is not surprising that the quantity would have a sizable negative correlation with generalization. The quantity I would consider to be the *main term* is the quantity *fro/spec*. Furthermore, the fact that this quantity (which appears in both Bartlett et al. and Arora et al.) performs somewhat worse than simply *sum-of-fro* is fairly interesting and possibly merits some discussion.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Fantastic Generalization Measures and Where to Find Them
Generalization of deep networks has been intensely researched in recent years, resulting in a number of theoretical bounds and empirically motivated measures. However, most papers proposing such measures only study a small set of models, leaving open the question of whether these measures are truly useful in practice. We present the first large scale study of generalization bounds and measures in deep networks. We train over two thousand CIFAR-10 networks with systematic changes in important hyper-parameters. We attempt to uncover potential causal relationships between each measure and generalization, by using rank correlation coefficient and its modified forms. We analyze the results and show that some of the studied measures are very promising for further research.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper addresses the well-known generalization problem for deep networks: why do these models successfully generalize even though they contain more parameters than the size of the training set? Recent works on this problem have proposed various measures of model complexity (that is, different from the number of parameters) with the claim that they (at least partially) capture the *true* capacity of deep networks in order to answer this question. The authors report the results of a large-scale, systematic study of the empirical effect of these complexity measures on the observed generalization behavior of deep networks. The empirical study of generalization behavior is inherently tricky, and the authors make a non-trivial contribution just by designing an experiment that can convincingly address this question. The authors give careful attention to various potential obstacles related to experimental design and statistical validity. The results of the authors* experiments are highly valuable to the research community. I recommend to accept. ********************************************************************************* Technical comments for the authors: -Please rewrite or elaborate on the first two paragraphs of section 6 for clarity. -It seems to make little sense to consider norm-based complexity measures that are not margin-normalized (or at least normalized in some other way). Please include a margin-normalized version of each such measure in Table 5; certain key quantities (e.g. *frob-distance*) have no corresponding margin-normalized entry. -The quantity *sum-of-frob/margin* is certainly a quantity of interest and it should be included in table 2 and possibly merits some discussion. -I disagree somewhat with the discussion of the spectral bound in section 7. In particular, I would not agree that the poor performance of *spec-orig* is surprising, because it contains an astronomically-sized *proof artifact* (as you call it). Namely, in this bound and in similar norm-based bounds, the term which is the product of the matrices in the network is (heuristically) unnecessary; in particular it (heuristically) ought to be able to be replaced with the network*s *average-case Lipschitz constant* (this is one of the issues which Arora et al. 2018 attempted to address; unfortunately formalizing an adequate notion of *average-case Lipschitz constant* can be quite cumbersome). This superfluous product of matrix norms is of course highly correlated with model size, so it is not surprising that the quantity would have a sizable negative correlation with generalization. The quantity I would consider to be the *main term* is the quantity *fro/spec*. Furthermore, the fact that this quantity (which appears in both Bartlett et al. and Arora et al.) performs somewhat worse than simply *sum-of-fro* is fairly interesting and possibly merits some discussion.
assistant
The paper addresses the well-known generalization problem for deep networks: why do these models successfully generalize even though they contain more parameters than the size of the training set? Recent works on this problem have proposed various measures of model complexity (that is, different from the number of parameters) with the claim that they (at least partially) capture the *true* capacity of deep networks in order to answer this question. The authors report the results of a large-scale, systematic study of the empirical effect of these complexity measures on the observed generalization behavior of deep networks. The empirical study of generalization behavior is inherently tricky, and the authors make a non-trivial contribution just by designing an experiment that can convincingly address this question. The authors give careful attention to various potential obstacles related to experimental design and statistical validity. The results of the authors* experiments are highly valuable to the research community. I recommend to accept. ********************************************************************************* Technical comments for the authors: -Please rewrite or elaborate on the first two paragraphs of section 6 for clarity. -It seems to make little sense to consider norm-based complexity measures that are not margin-normalized (or at least normalized in some other way). Please include a margin-normalized version of each such measure in Table 5; certain key quantities (e.g. *frob-distance*) have no corresponding margin-normalized entry. -The quantity *sum-of-frob/margin* is certainly a quantity of interest and it should be included in table 2 and possibly merits some discussion. -I disagree somewhat with the discussion of the spectral bound in section 7. In particular, I would not agree that the poor performance of *spec-orig* is surprising, because it contains an astronomically-sized *proof artifact* (as you call it). Namely, in this bound and in similar norm-based bounds, the term which is the product of the matrices in the network is (heuristically) unnecessary; in particular it (heuristically) ought to be able to be replaced with the network*s *average-case Lipschitz constant* (this is one of the issues which Arora et al. 2018 attempted to address; unfortunately formalizing an adequate notion of *average-case Lipschitz constant* can be quite cumbersome). This superfluous product of matrix norms is of course highly correlated with model size, so it is not surprising that the quantity would have a sizable negative correlation with generalization. The quantity I would consider to be the *main term* is the quantity *fro/spec*. Furthermore, the fact that this quantity (which appears in both Bartlett et al. and Arora et al.) performs somewhat worse than simply *sum-of-fro* is fairly interesting and possibly merits some discussion.