Optimal margin Distribution Network
Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.
 I consider that improving the generalization capability of neural networks on small dataset is an important line of research, and the method proposed here empirically provides great results. The proposed margin loss (Equation 1) is said to be "specially adapted for accelerating the convergence velocity of networks by [the authors]". I would like this statement to be explained better, or at least backed by empirical evidence. In the current state, I consider that the paper lacks an in-depth study of the properties of this handcrafted loss. Few is said on the benefits of having both a linear behavior for points inside the margin and a quadratic loss for far points. The impact of loss hyperparameters (r, .gamma,.mu) should be discussed thoughtfully; at some points in the paper, r and .gamma are referred as margin mean and margin variance parameters, but this interpretation is not explained. Moreover, almost nothing is said about .mu. By considering a simplified loss function, the provided PAC-Bayes generalization bound (Theorem 1) consider solely the flat loss region [r-.gamma, r+.gamma], but shed no light on the benefit of the hinge and quadratic parts. I conceive that this might be hard to study theoretically, but the authors should at least provide a empirical study of these. The empirical experiments show great evidence that the proposed method successfully improve generalization capability of neural networks on small datasets compared to classical methods. I appreciate the Inter/intra class variance study of Tables 2 and 3. I would like the mathematical expression of the "hinge loss" and the "soft hinge loss" models to be explicitly written (it is not clear in the text if the soft hinge uses an hyperparameter). In the same spirit of my above comments, I would like to see how each loss hyperparameters impacts the results, instead of having access solely to the parameter values selected by the validation process. Typos and minor comments: - Abstract: "And our ODN model also outperforms the other three loss models..." Which three loss models? - Section 3: "Specially, define L_0 as r=.theta..." I think it should be r=0 - Section 4.1: model-s => models - Page 7 (and elsewhere): Table. 2 => Table 2 - Please specify that "Xent" stands for cross-entropy - Figure 3: Please use larger font sizes - Proof of Lemma 2: Equation. 4 => Equation 4
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Optimal margin Distribution Network
Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PAC-Bayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 I consider that improving the generalization capability of neural networks on small dataset is an important line of research, and the method proposed here empirically provides great results. The proposed margin loss (Equation 1) is said to be "specially adapted for accelerating the convergence velocity of networks by [the authors]". I would like this statement to be explained better, or at least backed by empirical evidence. In the current state, I consider that the paper lacks an in-depth study of the properties of this handcrafted loss. Few is said on the benefits of having both a linear behavior for points inside the margin and a quadratic loss for far points. The impact of loss hyperparameters (r, .gamma,.mu) should be discussed thoughtfully; at some points in the paper, r and .gamma are referred as margin mean and margin variance parameters, but this interpretation is not explained. Moreover, almost nothing is said about .mu. By considering a simplified loss function, the provided PAC-Bayes generalization bound (Theorem 1) consider solely the flat loss region [r-.gamma, r+.gamma], but shed no light on the benefit of the hinge and quadratic parts. I conceive that this might be hard to study theoretically, but the authors should at least provide a empirical study of these. The empirical experiments show great evidence that the proposed method successfully improve generalization capability of neural networks on small datasets compared to classical methods. I appreciate the Inter/intra class variance study of Tables 2 and 3. I would like the mathematical expression of the "hinge loss" and the "soft hinge loss" models to be explicitly written (it is not clear in the text if the soft hinge uses an hyperparameter). In the same spirit of my above comments, I would like to see how each loss hyperparameters impacts the results, instead of having access solely to the parameter values selected by the validation process. Typos and minor comments: - Abstract: "And our ODN model also outperforms the other three loss models..." Which three loss models? - Section 3: "Specially, define L_0 as r=.theta..." I think it should be r=0 - Section 4.1: model-s => models - Page 7 (and elsewhere): Table. 2 => Table 2 - Please specify that "Xent" stands for cross-entropy - Figure 3: Please use larger font sizes - Proof of Lemma 2: Equation. 4 => Equation 4
assistant
 I consider that improving the generalization capability of neural networks on small dataset is an important line of research, and the method proposed here empirically provides great results. The proposed margin loss (Equation 1) is said to be "specially adapted for accelerating the convergence velocity of networks by [the authors]". I would like this statement to be explained better, or at least backed by empirical evidence. In the current state, I consider that the paper lacks an in-depth study of the properties of this handcrafted loss. Few is said on the benefits of having both a linear behavior for points inside the margin and a quadratic loss for far points. The impact of loss hyperparameters (r, .gamma,.mu) should be discussed thoughtfully; at some points in the paper, r and .gamma are referred as margin mean and margin variance parameters, but this interpretation is not explained. Moreover, almost nothing is said about .mu. By considering a simplified loss function, the provided PAC-Bayes generalization bound (Theorem 1) consider solely the flat loss region [r-.gamma, r+.gamma], but shed no light on the benefit of the hinge and quadratic parts. I conceive that this might be hard to study theoretically, but the authors should at least provide a empirical study of these. The empirical experiments show great evidence that the proposed method successfully improve generalization capability of neural networks on small datasets compared to classical methods. I appreciate the Inter/intra class variance study of Tables 2 and 3. I would like the mathematical expression of the "hinge loss" and the "soft hinge loss" models to be explicitly written (it is not clear in the text if the soft hinge uses an hyperparameter). In the same spirit of my above comments, I would like to see how each loss hyperparameters impacts the results, instead of having access solely to the parameter values selected by the validation process. Typos and minor comments: - Abstract: "And our ODN model also outperforms the other three loss models..." Which three loss models? - Section 3: "Specially, define L_0 as r=.theta..." I think it should be r=0 - Section 4.1: model-s => models - Page 7 (and elsewhere): Table. 2 => Table 2 - Please specify that "Xent" stands for cross-entropy - Figure 3: Please use larger font sizes - Proof of Lemma 2: Equation. 4 => Equation 4