The evaluation of robustness against adversarial manipulations of neural networks-based classifiers is mainly tested with empirical attacks as the methods for the exact computation, even when available, do not scale to large networks. We propose in this paper a new white-box adversarial attack wrt the l p -norms for p ∈ {1, 2, ∞} aiming at finding the minimal perturbation necessary to change the class of a given input. It has an intuitive geometric meaning, yields quickly high quality results, minimizes the size of the perturbation (so that it returns the robust accuracy at every threshold with a single run). It performs better or similarly to state-of-the-art attacks which are partially specialized to one l p -norm.
The authors propose a new gradient-based method (FAB) for constructing adversarial perturbations for deep neural networks. At a high level, the method repeatedly estimates the decision boundary based on the linearization of the classifier at a given point and projects to the closest "misclassified" example based on that estimation (similar to DeepFool). The authors build on this idea, proposing several improvements and evaluate their attack empirically against a variety of models.
I found the proposed method quite interesting and intuitive. All the improvements made to the core method are well-motivated and clearly explained, while the ablation experiments are relatively thorough.
However, I did find the presentation of experimental evidence quite misleading. 
Specifically, reporting mean accuracy over models, datasets, and epsilon constraints in Table 2 does not give the full picture. Going through the appendix tables, we can see the following:
-- The step size used for PGD is quite large---eps/4 for the L2 case---which is quite uncommon when using 150 iterations. Based on prior work and my own personal experience, a step size of 2 * eps / #steps (i.e., eps / 75) would seem more suitable. I wonder if this is the reason for PGD performing worse than FAB for large epsilon values on CIFAR10. The authors mention that they chose this parameter using grid search but do not provide concrete details.
-- The adversarially trained MNIST model of Madry et al. 2018 learns to use thresholding filters as the first layer (observed in the original paper). This causes issues for most gradient-based methods (e.g., PGD performs worse than the decision-based attack of Brendel et al. 2018, also observed in other prior work). While it is encouraging that FAB is robust to such gradient obfuscation, this is arguably not the ideal setting to compare gradient based methods (especially when averaging performance over models). 
-- For MNIST and Restricted IN, PGD performs comparably or even better than FAB (modulo larger epsilon values for which the large step size used could be an issue for PGD and the Linf-trained model with the thresholding filters).
-- For the L1-norm setting, EAD performs similarly or better compared to FAB (again modulo the Linf-trained model).
Based on these observations, I am not fully convinced that FAB outperforms PGD (for L2 and Linf) and EAD (for L1) by as much as Table 2 suggests.
Moreover, the runtime comparison performed in not exactly fair:
-- It is not clear how many restarts where included in the runtime of PGD. Its runtime should be in the same ballpark as FAB but the time reported is ~20x higher. 
-- PGD is known to produce quite accurate estimates when run with much fewer (say 15) steps. Thus in order to make a fair comparison one would also need to look at the entire #steps vs robust accuracy curve to get a better picture of the efficiency of these two methods. Choosing an arbitrary number of steps for each method is not very enlightening.
-- It is not necessary to run PGD 5 times to evaluate the robust accuracy at 5 thresholds. One can perform binary search for each input in order to find the smallest epsilon for which a misclassification can be found. This will result in at most 3 (sometimes 2) evaluations per point (instead of 5).
Despite these shortcomings of the experimental evaluation, I still believe that the paper has merit. After all, the method is clean and well-motivated,  performs comparably to the best of PGD and EAD in a variety of settings, and is robust to a certain degree of gradient masking. In that sense, it could potentially be a valuable contribution and could be of interest to a subset of the adversarial ML community.
In the sense, while my initial stance is to recommend (weak) rejection, I would be open to increasing my score and recommending (weak) acceptance should my concerns be addressed.
UPDATE: I appreciate the response and the additional experiments performed by the authors. The authors have addressed my concerns in their response. I am increasing my score to a weak accept.
One thing that would be nice to add in the next version of the manuscript is a note inviting the reader to consider the appendix tables since average robust accuracy can be inconclusive.