Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unseen corruptions and perturbations encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AUGMIX, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unseen corruptions. AUGMIX significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.
This paper proposes a method called AugMix, which is intended to improve model robustness to data distribution shift. AugMix appears fairly simple to implement. Several new images are created by augmenting an original image through chains of sequentially applied transformations (the "Aug" part of AugMix), then the augmented images are combined together, along with the original image, via a weighted sum (the "Mix" part of AugMix). Additionally, a Jensen-Shannon Divergence consistency loss is applied during training to encourage the model to make similar predictions for all augmented variations of a single image. This technique is shown to achieve state-of-the-art performance on standard robustness benchmarks without loss of clean test accuracy, and is also shown to improve calibration of model confidence estimates.
Overall, I would tend to vote for accepting this paper. The method is simple yet effective, the paper is very well written and easy to follow, and experiments are extensive and, for the most part, convincing.
Questions:
1) The one main concern I have with the training procedure is the amount of time the models were trained for. It is known that model trained with aggressive data augmentation schemes often require much longer training than normal in order to fully benefit from the stronger augmentations. For example, AutoAugment trains ImageNet models for 270 epochs [1], while CutMix trains for 300 epochs [2]. However, the ImageNet experiments in this paper claim to follow the procedure outlined in [3], which only trains for 90 epochs. This is reflected in the clean test accuracy, where AutoAugment only appears to provide a 0.3% gain over standard training, while we might expect 1.3% improvement (according to [2]). The AllConvNet and WideResNet in CIFAR-10 and CIFAR-100 experiments were also trained for only 100 epochs each, where 200 is more conventional. Again this shows in the reported numbers: on WideResNet for CIFAR-10, Mixup only has a 0.3% gain were as we might expect 1% improvement instead [4], and AutoAugment has 0.4% improvement, were as we might expect 1.3% gain if trained longer [1]. My question then is, how much does training time affect results? Do AugMix, and other techniques such as Mixup, CutMix, and AutoAugment, achieve better robustness when models are trained for longer, or do they become more brittle as training time is extended? 
2) For the Jensen-Shannon divergence consistency, how much worse does it perform when using JS(Porig;Paugmix1) versus JS(Porig;Paugmix1;Paugmix2)? What might cause this behaviour?
3) Patch Gaussian is changed to Patch Uniform to avoid overlap with corruptions in ImageNet-C. How does Patch Uniform compare to Patch Gaussian in terms of performance for non-Gaussian noise corruptions?
4) How does AugMix perform as an augmentation technique in terms of clean test accuracy compared to other SOTA techniques? Is there a trade-off between clean test accuracy and robustness, or does AugMix improve performance in both domains? Can AugMix be combined with other augmentation techniques or does this destroy robustness properties?
Things to improve the paper that did not impact the score:
5) It would be nice if the best result in each column could be bolded in Tables 2-4.
References:
[1] Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. "Autoaugment: Learning augmentation policies from data." CVPR (2019).
[2] Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. "Cutmix: Regularization strategy to train strong classifiers with localizable features." ICCV (2019).
[3] Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. "Accurate, large minibatch sgd: Training imagenet in 1 hour." arXiv preprint arXiv:1706.02677 (2017).
[4] Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. "mixup: Beyond empirical risk minimization." ICLR (2018).