While deep learning has led to remarkable results on a number of challenging problems, researchers have discovered a vulnerability of neural networks in adversarial settings, where small but carefully chosen perturbations to the input can make the models produce extremely inaccurate outputs. This makes these models particularly unsuitable for safety-critical application domains (e.g. self-driving cars) where robustness is extremely important. Recent work has shown that augmenting training with adversarially generated data provides some degree of robustness against test-time attacks. In this paper we investigate how this approach scales as we increase the computational budget given to the defender. We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model. Crucially, we show that it is the adversarial training of the ensemble, rather than the ensembling of adversarially trained models, which provides robustness.
Summary. The paper considers the robustness of neural nets against adversarial attacks. More precisely, the authors experimentally investigate the robustness of ensembles of neural nets. They empirically show that adversarially trained ensembles of 2 neural nets are more robust than ensembles of 2 adversarially trained neural nets.
Pros.
* Robustness of neural nets is a challenging problem of interest for ICLR
* The paper is easy to read
* Experimental results compare different algorithms for 2 neural nets
Cons.
* The study is experimental
* It is limited to gradient-based attacks
* It is limited to ensembles of size 2
* The Ensemble2Adv is a single NN model and not an ensemble model. 
Evaluation.
The problem is significant and the use of ensemble methods for robustness against adversarial attacks is a promising line of research. The experimental study in this paper opens new lines of research in this direction. But, in my opinion, the paper is not ready for publication at ICLR. Detailed comments follow but the study is limited to k=2; the main finding is limited to the comparison between bagging two adversarially trained neural nets (SeparateEnsemble2Adv) and learning adversarially the average of two neural nets (Ensemble2adv). In my opinion, Ensemble2adv is a single model of double size and not an ensemble model thus somehow contradicting the main claim of the paper.
Detailed comments.
* Introduction, end of §2, it is said that non-gradient based attacks are still effective. But in the sequel you only consider gradient-based attacks and never discussed this question.
* Introduction, contributions, it should be made clear at the beginning of the paper that you will consider ensembles of size 2 and only gradient-based attacks.
* Section 2. The momentum-based attack should be cited and could be considered. "Boosting adversarial attacks with momentum, Dong et al, CVPR18"
* Section 3, §2, the discussion on ensemble methods is unprecise. Ensemble methods have different objectives. For instance, Bagging-like methods  aim at reducing the generalization error while others as Boosting aim at augmenting the capacity of individual models.
* Section 3. Here is my main concern on this paper. The classical method would be bagging of neural nets with different initializations. The neural nets could be adversarially trained. This would lead to the so-called SeparateEnsemble2Adv. Here, the authors consider another method. Their method can be viewed as k(=2) copies of the same neural network with different initializations and an additional layer computing the average of the k(=2) outputs. Then adversarially learn the obtained model which leads to the so-called Ensemble2Adv algorithm. This algorithm is not an ensemble method as such. In my opinion for k=2, it is equivalent to doubling the size of a neural net, adding averaging of the outputs, and adversarially training the obtained neural net.
* Note recent advances in ensemble NNs with papers such as Averaging weights leads to wider optima ..., Izmailov et al, UAI18; Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, Garipov et al, arXiv:1802.10026
* Section 4.1. Here comes the limitation k=2. The case k=4 is considered in table 1 but is not discussed elsewhere in the paper.
* Section 4.1. I am not convinced by DoubleAdv. It is one way of doubling the size of a neural net but I am not convinced that this is the more efficient. As said before, in my opinion, Ensemble2Adv is another way for doubling the size. And many more should exist.
* Section 5. In my opinion, the main comparisons should concern SeparateEnsemble2Adv and Ensemble2Adv. Also other methods doubling the size should be considered. 
* Section 5. For k greater than 2, SeparateEnsemblekAdv should be the better method because the adversarial learning phase could be easily parallelized.
* I am not convinced by the discussion in Section 6.
* Typos. and -> an l-13, p5; IFGSM5, l-19 p6; then l-6 p7; to due l-6 p9
* Biblio. Please give complete references