Adversarial training is one of the main defenses against adversarial attacks. In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on ImageNet, which reveals two intriguing properties. First, we study the role of normalization. Batch Normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This twodomain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness. Second, we study the role of network capacity. We find our so-called "deep" networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.
This paper reveals some interesting properties of neural networks when trained adversarially at ImageNet scale. The total cost of the experiments is quite impressive, therefore the results are valuable references. With extensive experiments, the authors reveals two intriguing properties of neural networks when trained adversarially, and devotes most of the paper to studying the first one, i.e., why networks with Batch Normalization cannot achieve high robustness when both clean and adversarial images are used for training. I am not fully convinced by this point, but I do think the second point is very interesting. Different from previous work arguing that more data is needed for improving adversarially robust generalization, this paper shows that adversarial robustness can be improved consistently just by making ResNet deeper. 
I tend to accept this paper for the valuable results and the discovery of the positive correlation between network capacity and adversarial robustness, but I am not fully convinced by the explanations for the problems with BN. I hope the authors could address my concerns before I can be more confident about my decision.
To me, it seems the correct way to do adversarial training is to only use adversarial samples, since we are trying to minimize the maximum risk inside a norm ball around the clean sample (Madry et al. 2018). All the experiments with clean images in the objective are consistently worse than training only on adversarial samples under the same setting in this paper. It therefore becomes not that important to study the effect of Batch Normalization on training with both clean and adversarial images.
Also, I am a little bit unconvinced that the running mean and variance of BN is the cause of bad performance for the mixed training. First, I want to know the standard and adversarial accuracies of the network with GN in the "100% adv + 0% clean" setting. It seems missing in the paper. Despite being much better than BN in the "100% adv + 100% clean" setting, it is not sure whether it is caused by the improvement in some other property (e.g., capacity) from GN. If the robust accuracy with "100% adv + 0% clean" is also much higher with GN than BN (seems unlikely though), then replacing GN with BN does not solve the problem and it is still the objective to blame.
I am also not fully convinced by the experiments with separated BN parameters for clean and adversarial samples. By doing so, the network is actually trained to approximate its behavior in the "100% adv + 0% clean" setting. Since the adversary is maximizing the loss, the gradient (of the conv layers) from adversarial samples will dominate the total gradient in the "MBN 100% adv + 100% clean" setting, and the "MBN_{adv}" network will be trained similarly as the network trained in the "100% adv + 0% clean" setting. This explains why the adversarial accuracy can be very close to "100% adv + 0% clean" but still lower.  Comparing results (under different ratios of clean samples) for networks without BN is very important and much more effective at explaining the phenomenon than trying to make BN work.
It would also be great if the authors could provide some curve showing the tendency of the running variance of BN. Sec 4.3 does make lots of sense from a practical aspect, i.e., fixing mean and var for training in the last 10 epochs could improve the results using same number of epochs, but what if we just train more epochs? Will the variance converge in just tens of epochs?
Finally, though it seems that deeper networks are more robust,  the robustness might be a misconception caused by gradient vanishing. Could the authors provide the average gradient norms on the correctly-classified images (remaining correct after attack) in the first step of PGD attacks for the models in Figure 7? If deeper networks indeed have much smaller gradient norm, could the authors try scaling the loss by some factor to make the attacks stronger?
===========================================================
I am not fully convinced by your explanations. Could you give the results for the gradients, or use the loss function from CW attack + PGD to report the robustness of the deeper models? I am not sure which type of residual connection you are using, but if you are using the "original" version as mentioned in this paper [1], then it does not necessarily avoid gradient vanishing, since if ReLU is deactivated, the gradient will be brought to zero. Another possible cause for the possible misconception of robustness might be caused by the saturated cross entropy loss, which could also give 0 gradients but could be verified by switching to the margin loss as in the CW attack. I have encountered such cases when I achieved non-zero robust accuracy (evaluated with PGD on cross entropy loss, and the robust accuracy is similar to your improvement from ResNet152 to ResNet638) on a naturally trained network as I increased the number of layers, but when I switched to CW loss (or margin loss), the robust accuracy goes to zero. Even for resnet, when you multiply the number of layers by ~4 as done in the paper, such phenomenon will actually happen.
[1] He, Kaiming, et al. "Identity mappings in deep residual networks." European conference on computer vision. Springer, Cham, 2016.
=====================================================================
Raising my score. Found a recent good paper based on the findings of this paper.