Generalization Puzzles in Deep Networks
In the last few years, deep learning has been tremendously successful in many applications. However, our theoretical understanding of deep learning, and thus the ability of providing principled improvements, seems to lag behind. A theoretical puzzle concerns the ability of deep networks to predict well despite their intriguing apparent lack of generalization: their classification accuracy on the training set is not a proxy for their performance on a test set. How is it possible that training performance is independent of testing performance? Do indeed deep networks require a drastically new theory of generalization? Or are there measurements based on the training data that are predictive of the network performance on future data? Here we show that when performance is measured appropriately, the training performance is in fact predictive of expected performance, consistently with classical machine learning theory.
Summary: This paper investigates the question of generalization in deep networks. The main contribution is to that show train loss can be predictive of the test loss when you normalize the networks, for over-parameterized network using ReLu activation functions. Authors perform a set of experiments on CIFAR10/CIFAR100 and MNIST to validate their claim. Comments: Overall the paper main message is clear, however some points/formatting in the paper could be improved: - Authors state in section 2 that they use CIFAR10, MNIST, CIFAR100. While the results on CIFAR100/MNIST are in the appendix, I did not find references/descriptions of those results in the paper main body. - Section 2 refers to equation 7,8 several times. Those equations are only defined in the appendix. It seems that authors should refer to equation 3 and 4 instead? - Authors refers to Figure 8 in section 2 without describing it. It is unclear why Figure 8 is needed here? - Section 2 claims that the Rademacher complexity in the experiment is found to be small for the normalized network. But it is unclear which data/figure support this claim? - Section 3 refers to figure 9 which appears unrelated (and figure 9 caption refers to figure 10). I think section 3 should refers to figure 4 instead. - I did not find any reference to figure 3 in the main text - In section 4, I find the following part ‘generalization for the classification error and for the unnormalized cross-entropy is attained for much larger N […] N must be significantly larger than the number of parameters...’ a bit unclear. Could you elaborate given that overparameterized networks can achieve good validation test - Abstract claims that ‘we show that when the performance is measured appropriately […] training performance is predictive of expected performance’. In practice, the paper assumes that the networks are overparameterized and uses non-linearity which satisfies the positive homogeneity property. While those limitation are clearly stated in the section, it would be nice to make that clear in the abstract/introduction as well - In section 2, authors say that ‘most datasets are separable by overparameterization’. It would be nice to elaborate on that or add a citation justifying this claim. In addition, while the paper clearly show that the normalized train cross-entropy can predict the performance of the of the normalized test cross-entropy, it is unclear if the normalized train cross-entropy is predictive of the test classification error ? While the paper writing and clarity could be improved, I think the paper makes interesting observation, I therefore lean toward acceptance.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Generalization Puzzles in Deep Networks
In the last few years, deep learning has been tremendously successful in many applications. However, our theoretical understanding of deep learning, and thus the ability of providing principled improvements, seems to lag behind. A theoretical puzzle concerns the ability of deep networks to predict well despite their intriguing apparent lack of generalization: their classification accuracy on the training set is not a proxy for their performance on a test set. How is it possible that training performance is independent of testing performance? Do indeed deep networks require a drastically new theory of generalization? Or are there measurements based on the training data that are predictive of the network performance on future data? Here we show that when performance is measured appropriately, the training performance is in fact predictive of expected performance, consistently with classical machine learning theory.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: This paper investigates the question of generalization in deep networks. The main contribution is to that show train loss can be predictive of the test loss when you normalize the networks, for over-parameterized network using ReLu activation functions. Authors perform a set of experiments on CIFAR10/CIFAR100 and MNIST to validate their claim. Comments: Overall the paper main message is clear, however some points/formatting in the paper could be improved: - Authors state in section 2 that they use CIFAR10, MNIST, CIFAR100. While the results on CIFAR100/MNIST are in the appendix, I did not find references/descriptions of those results in the paper main body. - Section 2 refers to equation 7,8 several times. Those equations are only defined in the appendix. It seems that authors should refer to equation 3 and 4 instead? - Authors refers to Figure 8 in section 2 without describing it. It is unclear why Figure 8 is needed here? - Section 2 claims that the Rademacher complexity in the experiment is found to be small for the normalized network. But it is unclear which data/figure support this claim? - Section 3 refers to figure 9 which appears unrelated (and figure 9 caption refers to figure 10). I think section 3 should refers to figure 4 instead. - I did not find any reference to figure 3 in the main text - In section 4, I find the following part ‘generalization for the classification error and for the unnormalized cross-entropy is attained for much larger N […] N must be significantly larger than the number of parameters...’ a bit unclear. Could you elaborate given that overparameterized networks can achieve good validation test - Abstract claims that ‘we show that when the performance is measured appropriately […] training performance is predictive of expected performance’. In practice, the paper assumes that the networks are overparameterized and uses non-linearity which satisfies the positive homogeneity property. While those limitation are clearly stated in the section, it would be nice to make that clear in the abstract/introduction as well - In section 2, authors say that ‘most datasets are separable by overparameterization’. It would be nice to elaborate on that or add a citation justifying this claim. In addition, while the paper clearly show that the normalized train cross-entropy can predict the performance of the of the normalized test cross-entropy, it is unclear if the normalized train cross-entropy is predictive of the test classification error ? While the paper writing and clarity could be improved, I think the paper makes interesting observation, I therefore lean toward acceptance.
assistant
Summary: This paper investigates the question of generalization in deep networks. The main contribution is to that show train loss can be predictive of the test loss when you normalize the networks, for over-parameterized network using ReLu activation functions. Authors perform a set of experiments on CIFAR10/CIFAR100 and MNIST to validate their claim. Comments: Overall the paper main message is clear, however some points/formatting in the paper could be improved: - Authors state in section 2 that they use CIFAR10, MNIST, CIFAR100. While the results on CIFAR100/MNIST are in the appendix, I did not find references/descriptions of those results in the paper main body. - Section 2 refers to equation 7,8 several times. Those equations are only defined in the appendix. It seems that authors should refer to equation 3 and 4 instead? - Authors refers to Figure 8 in section 2 without describing it. It is unclear why Figure 8 is needed here? - Section 2 claims that the Rademacher complexity in the experiment is found to be small for the normalized network. But it is unclear which data/figure support this claim? - Section 3 refers to figure 9 which appears unrelated (and figure 9 caption refers to figure 10). I think section 3 should refers to figure 4 instead. - I did not find any reference to figure 3 in the main text - In section 4, I find the following part ‘generalization for the classification error and for the unnormalized cross-entropy is attained for much larger N […] N must be significantly larger than the number of parameters...’ a bit unclear. Could you elaborate given that overparameterized networks can achieve good validation test - Abstract claims that ‘we show that when the performance is measured appropriately […] training performance is predictive of expected performance’. In practice, the paper assumes that the networks are overparameterized and uses non-linearity which satisfies the positive homogeneity property. While those limitation are clearly stated in the section, it would be nice to make that clear in the abstract/introduction as well - In section 2, authors say that ‘most datasets are separable by overparameterization’. It would be nice to elaborate on that or add a citation justifying this claim. In addition, while the paper clearly show that the normalized train cross-entropy can predict the performance of the of the normalized test cross-entropy, it is unclear if the normalized train cross-entropy is predictive of the test classification error ? While the paper writing and clarity could be improved, I think the paper makes interesting observation, I therefore lean toward acceptance.