ON BREIMAN’S DILEMMA IN NEURAL NETWORKS: SUCCESS AND FAILURE OF NORMALIZED MARGINS
A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overfitting by increasing the robustness. Yet Breiman shows a dilemma (Breiman, 1999) that a uniform improvement on margin distribution emph{does not} necessarily reduces generalization error. In this paper, we revisit Breiman*s dilemma in deep neural networks with recently proposed normalized margins using Lipschitz constant bound by spectral norm products. With both simplified theory and extensive experiments, Breiman*s dilemma is shown to rely on dynamics of normalized margin distributions, that reflects the trade-off between model expression power and data complexity. When the complexity of data is comparable to the model expression power in the sense that training and test data share similar phase transitions in normalized margin dynamics, two efficient ways are derived via classic margin-based generalization bounds to successfully predict the trend of generalization error. On the other hand, over-expressed models that exhibit uniform improvements on training normalized margins may lose such a prediction power and fail to prevent the overfitting.
 The submission explores Breiman*s dilemma: training margin is not always a good predictor of test error. In particular, the authors show that: - For under-parametrized CNNs, the training prediction margin is a good predictor of the test error. - For over-parametrized CNNs, the training prediction margin is not a good predictor of the test error. Throughout the submission, I suspect that the authors compute the "functional margin", that is, the difference between the largest label score and the second largest score, for correctly classified examples. Functional margins ignore the smoothness of the underlying function, a critical factor for generalization. For instance, the function f(x) = 1[x > 0] has large functional margin, but any perturbation around the x-origin would drastically change the prediction. For this reason, I think the authors should consider the "geometrical margin" instead, which is unfortunately difficult to compute for general neural networks. Their theory tries to reflect on this issue by using spectrally-normalized bounds, but the practice ignores this issue completely (as far as I can tell). Therefore, we may be looking at the wrong statistic to predict generalization error. Is Breiman dilemma solved by re-defining margin properly? Geometrical margin can be computed in closed-form for linear classifiers, so perhaps this would be a first step in this investigation.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
ON BREIMAN’S DILEMMA IN NEURAL NETWORKS: SUCCESS AND FAILURE OF NORMALIZED MARGINS
A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overfitting by increasing the robustness. Yet Breiman shows a dilemma (Breiman, 1999) that a uniform improvement on margin distribution emph{does not} necessarily reduces generalization error. In this paper, we revisit Breiman*s dilemma in deep neural networks with recently proposed normalized margins using Lipschitz constant bound by spectral norm products. With both simplified theory and extensive experiments, Breiman*s dilemma is shown to rely on dynamics of normalized margin distributions, that reflects the trade-off between model expression power and data complexity. When the complexity of data is comparable to the model expression power in the sense that training and test data share similar phase transitions in normalized margin dynamics, two efficient ways are derived via classic margin-based generalization bounds to successfully predict the trend of generalization error. On the other hand, over-expressed models that exhibit uniform improvements on training normalized margins may lose such a prediction power and fail to prevent the overfitting.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The submission explores Breiman*s dilemma: training margin is not always a good predictor of test error. In particular, the authors show that: - For under-parametrized CNNs, the training prediction margin is a good predictor of the test error. - For over-parametrized CNNs, the training prediction margin is not a good predictor of the test error. Throughout the submission, I suspect that the authors compute the "functional margin", that is, the difference between the largest label score and the second largest score, for correctly classified examples. Functional margins ignore the smoothness of the underlying function, a critical factor for generalization. For instance, the function f(x) = 1[x > 0] has large functional margin, but any perturbation around the x-origin would drastically change the prediction. For this reason, I think the authors should consider the "geometrical margin" instead, which is unfortunately difficult to compute for general neural networks. Their theory tries to reflect on this issue by using spectrally-normalized bounds, but the practice ignores this issue completely (as far as I can tell). Therefore, we may be looking at the wrong statistic to predict generalization error. Is Breiman dilemma solved by re-defining margin properly? Geometrical margin can be computed in closed-form for linear classifiers, so perhaps this would be a first step in this investigation.
assistant
 The submission explores Breiman*s dilemma: training margin is not always a good predictor of test error. In particular, the authors show that: - For under-parametrized CNNs, the training prediction margin is a good predictor of the test error. - For over-parametrized CNNs, the training prediction margin is not a good predictor of the test error. Throughout the submission, I suspect that the authors compute the "functional margin", that is, the difference between the largest label score and the second largest score, for correctly classified examples. Functional margins ignore the smoothness of the underlying function, a critical factor for generalization. For instance, the function f(x) = 1[x > 0] has large functional margin, but any perturbation around the x-origin would drastically change the prediction. For this reason, I think the authors should consider the "geometrical margin" instead, which is unfortunately difficult to compute for general neural networks. Their theory tries to reflect on this issue by using spectrally-normalized bounds, but the practice ignores this issue completely (as far as I can tell). Therefore, we may be looking at the wrong statistic to predict generalization error. Is Breiman dilemma solved by re-defining margin properly? Geometrical margin can be computed in closed-form for linear classifiers, so perhaps this would be a first step in this investigation.