Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping high accuracy on clean inputs comparable to the SCE loss with little extra computation.
[Post rebuttal: The rebuttal addresses most of my concerns. I have updated my score.]
This paper points out some limitation of the traditional softmax cross-entropy (SCE) loss, e.g., causing indirect and unexpected supervisory signals on the learned features so that the points with low loss values tend to spread over the space sparsely. To remedy this, the authors then proposes the max-mahalanobis center (MMC) loss. By analyzing the sample density, it is proved that the MMC loss has a sample density proportional to the number of data samples in each class. This means MMC loss induces higher feature densities than the SCE loss, thus is expected to be more robust under adverssarial attacks.
I find the paper fairly well written in general, and the arguments are well supported by the development of the theoretical results. There are, however, several questions raised by the reviewer.
One question is that by looking at the definition (8), it seems the MMC loss essentially tries to move data from different classes apart. This seems to be quite similar to the max margin loss. I would expect the authors differentiate the differences between these two. Consequently, I would also like to see empirical comparisons between these two losses.
Another question is even if you have Theorem 2, the transfer from Theorem 2 to robustness does not seem direct, e.g., how do you guarantee when the density form in eq.9 is better than that in eq.7 for robustness?
Then when looking back at the sample density definition in eq. (2). I wonder how the Vol is defined to guarantee eq.2 is a valid density function? e.g., to guarantee the integration equals to 1.
In terms of experiments, since it is claimed that the proposed loss adds little computation cost compared to the SCE loss, I think it is better to include running time comparison in the results.
The last two lines in page 8, what are unseen attacks? From the experiments, it seems that it means the PGD with different steps. I think the authors  should be careful to use the term, because some literature use the term *different attacks* to refer to attacks with different *norms*.
Second line below Table 5: it is said the MMC loss keeps state-of-the-art performance on clean data. However, by looking at Table 4, the state-of-the-art is obvious SCE. Why did you say that?
Others:
1. S_k,\tilde{k}^2 is undefined.
2. How do you solve the minmax problem below eq.8? I guess you use the algorithm in Appendix B.1? Please clarify.