Recent research about margin theory has proved that maximizing the minimum margin like support vector machines does not necessarily lead to better performance, and instead, it is crucial to optimize the margin distribution. In the meantime, margin theory has been used to explain the empirical success of deep network in recent studies. In this paper, we present ODN (the Optimal margin Distribution Network), a network which embeds a loss function in regard to the optimal margin distribution. We give a theoretical analysis for our method using the PACBayesian framework, which confirms the significance of the margin distribution for classification within the framework of deep networks. In addition, empirical results show that the ODN model always outperforms the baseline cross-entropy loss model consistently across different regularization situations. And our ODN model also outperforms the cross-entropy loss (Xent), hinge loss and soft hinge loss model in generalization task through limited training data.
I consider that improving the generalization capability of neural networks on small dataset is an important line of research, and the method proposed here empirically provides great results.
The proposed margin loss (Equation 1) is said to be "specially adapted for accelerating the convergence velocity of networks by [the authors]". I would like this statement to be explained better, or at least backed by empirical evidence. In the current state, I consider that the paper lacks an in-depth study of the properties of this handcrafted loss. Few is said on the benefits of having both a linear behavior for points inside the margin and a quadratic loss for far points. The impact of loss hyperparameters (r, \gamma,\mu) should be discussed thoughtfully; at some points in the paper, r and \gamma are referred as margin mean and margin variance parameters, but this interpretation is not explained. Moreover, almost nothing is said about \mu.
By considering a simplified loss function, the provided PAC-Bayes generalization bound (Theorem 1) consider solely the flat loss region [r-\gamma, r+\gamma], but shed no light on the benefit of the hinge and quadratic parts. I conceive that this might be hard to study theoretically, but the authors should at least provide a empirical study of these. 
The empirical experiments show great evidence that the proposed method successfully improve generalization capability of neural networks on small datasets compared to classical methods. I appreciate the Inter/intra class variance study of Tables 2 and 3. I would like the mathematical expression of the "hinge loss" and the "soft hinge loss" models to be explicitly written (it is not clear in the text if the soft hinge uses an hyperparameter). In the same spirit of my above comments, I would like to see how each loss hyperparameters impacts the results, instead of having access solely to the parameter values selected by the validation process.
Typos and minor comments:
- Abstract: "And our ODN model also outperforms the other three loss models..." Which three loss models?
- Section 3: "Specially, define L_0 as r=\theta..." I think it should be r=0
- Section 4.1: model-s => models
- Page 7 (and elsewhere): Table. 2 => Table 2
- Please specify that "Xent" stands for cross-entropy
- Figure 3: Please use larger font sizes
- Proof of Lemma 2: Equation. 4 => Equation 4