Deep networks often perform well on the data distribution on which they are trained, yet give incorrect (and often very confident) answers when evaluated on points from off of the training distribution. This is exemplified by the adversarial examples phenomenon but can also be seen in terms of model generalization and domain shift. Ideally, a model would assign lower confidence to points unlike those from the training distribution. We propose a regularizer which addresses this issue by training with interpolated hidden states and encouraging the classifier to be less confident at these points. Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes. This has a major advantage in that it avoids the underfitting which can result from interpolating in the input space. We prove that the exact condition for this problem of underfitting to be avoided by Manifold Mixup is that the dimensionality of the hidden states exceeds the number of classes, which is often the case in practice. Additionally, this concentration can be seen as making the features in earlier layers more discriminative. We show that despite requiring no significant additional computation, Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples.
The paper proposes a novel method called Manifold Mixup, which linearly interpolating (with a careful selected mixing ratio) two feature maps in latent space as well as their labels during training, aiming at regularizing deep neural networks for better generalization and robust to adversarial attacks. The authors experimentally show that networks with Manifold Mixup as regularizer can improve accuracy for both supervised and semi-supervised learning, are robust to adversarial attacks, and obtain promising results on Negative Log-Likelihood on held out samples. 
The paper is well written and easy to follow. Various experiments are conducted to support the contributions of the paper.  Nevertheless, the technical novelty seems a bit weak to me. The method basically moves the interpolating process from input space as in MixUp to randomly selected hidden states. More importantly, some of the paper’s claims are not very convincing to me in its current form.
Major remarks:
1.	The authors suggest that Mixup can suffer from interpolations intersecting with a real sample, but how Manifold Mixup can avoid this issue is not very clear to me. 
The authors theoretically prove that with the proposed training cost in Manifold Mixup, the representation for each class will lie on a subspace of dimension dim (h) –d +1 (h and d are the hidden dimension and number of classes, respectively). I did not get the idea of how such dimension reduction relates to the ‘’flattening’’ of the manifold and in particular how such representations (representations for each class “concentrating into local regions”) can avoid the class collision issues as that in Mixup.
Experimentally, from Figures 3 and 4, it seems the class collision issue could be worse than that of Mixup. For example, for mixing ratio of 0.6 (meaning the created image has almost half labels from the two original images), MixUp clearly shows, for instance in the second row, that there are two overlapped images (Horse and Plane), but Manifold Mixup seems to have only the Plane in the mixed image with a soft label. 
2.	The observations of mixing in the hidden space is better than mixing in the input space seem to contradictive to the observations by Mixup, it would be very useful if the paper can make that much clear to the readers. I would suggest that the authors fully compare with MixUp in the supervised learning tasks, namely using all the datasets (including ImageNet) and networks architectures used in MixUp for supervised learning. In this way, the paper would be much more convincing because the proposed method is so close to MixUp and the observation here is contradictive.
3.	I wonder how sensitive is the parameter Alpha in Manifold Mixup. For example, how the mixing rate Alpha impacts the results for NLL and Semi-supervised learning in section 5.2? 
4.	It would be useful to also present the results for SVHN for supervised learning since the Cifar10 and Cifar100 datasets are similar, and the authors have already used SVHN for other task in the paper.
Minor remarks:
1.	In Table2, the result from AdaMix seems missed.
2.	Why not using Cifar100, but with a new dataset SVHN for the semi-supervised learning in section 5.2?
3.	In related work, regarding regularizing deep networks by perturbing the hidden states, the proposed method may relate to AgrLearn (Guo et al., Aggregated Learning: A Vector Quantization Approach to Learning with Neural Networks) as well.