Neural networks could misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset. In this work, we study the binary classification of linearly separable datasets and show that linear classifiers could also have decision boundaries that lie close to their training dataset if cross-entropy loss is used for training. In particular, we show that if the features of the training dataset lie in a low-dimensional affine subspace and the cross-entropy loss is minimized by using a gradient method, the margin between the training points and the decision boundary could be much smaller than the optimal value. This result is contrary to the conclusions of recent related works such as , and we identify the reason for this contradiction. In order to improve the margin, we introduce differential training, which is a training paradigm that uses a loss function defined on pairs of points from each class. We show that the decision boundary of a linear classifier trained with differential training indeed achieves the maximum margin. The results reveal the use of cross-entropy loss as one of the hidden culprits of adversarial examples and introduces a new direction to make neural networks robust against them.
Due to the large variance in reviewer scores, I was asked to give this additional review.
Background: [Soudry et al. 2018] showed that the iterates of gradient descent, when optimizing logistic regression on separable data, converge to the L2 max-margin (SVM) solution for homogeneous linear separators (without bias). These results were later extended to other models and optimization methods.
This paper has two main results:
1)	It clarifies that the results of [Soudry et al. 2018] do not apply to logistic regression when the linear separator has a bias term (“b”). This is because the homogenous max margin solution in the extended [x,1] space is not the same as non-homogeneous max margin solution in the original space: the first has a penalty on the size of the bias term, i.e.
min_{w,b} ||w||^2 + b^2 s.t. y_n (w’x_n+b) >= 1
, while the latter does not: 
min_{w,b} ||w||^2  s.t. y_n(w’x_n+b) >= 1
2)	It suggests using differential training to correct this issue.
However, I do not believe these contributions are enough for a publication in ICLR. First, (2) is simply a combination of two known results, as mentioned by Reviewer 2. Second, though I commend the authors for pointing out (1), I do not feel this by itself warrants a publication, for the following reasons:
a) It is very simple to explain (1) in only a few lines (as I did above). Therefore, it would be more informative just to write (1) as a comment on the original paper (the ICLR 2018 forum is still open), not as a completely new publication. For me, all the numerical demonstrations and examples of this simple issue did not add much.
b)	Regularizing the bias term usually does not make a significant difference to the sample complexity (see the end of section 15.1.1 in the textbook “Understanding Machine Learning: From Theory to Algorithms” by Shai Shalev Shwartz.). Furthermore, the main motivation behind [Soudry et al. 2018] was to explain implicit bias and generalization in deep networks, where there such max-margin results (which penalize all the parameters) could be used to derive generalization bounds (e.g., https://arxiv.org/abs/1810.05369).
c)	Lastly, the authors here say that “the solution obtained by cross-entropy minimization is different from the SVM solution”. This (as well as the title and abstract) may mislead the readers to think there is something wrong in the proofs of [Soudry et al. 2018] and later papers, and that logistic regression does not converge to the max-margin solution for homogeneous linear separators. However, the max-margin solution for homogeneous linear separators is also called the “max margin” or SVM solution (just for a different family). For example, see the previous paper on the topic [“Margin Maximizing Loss Functions”, Rosset et al. 2004] or section 15.1.1 in the textbook “Understanding Machine Learning: From Theory to Algorithms” by Shai Shalev Shwartz.  As I see it, the only issue in [Soudry et al. 2018] is the sentence “A bias term could be added in the usual way, extending x_n by an additional ’1’ component." which is confusing since it cannot be applied directly to the SVM solution. The authors should aim to pinpoint this issue, and clarify their phrasing to avoid such confusions.