Depending on how much information an adversary can access to, adversarial attacks can be classified as white-box attack and black-box attack. In both cases, optimization-based attack algorithms can achieve relatively low distortions and high attack success rates. However, they usually suffer from poor time and query complexities, thereby limiting their practical usefulness. In this work, we focus on the problem of developing efficient and effective optimization-based adversarial attack algorithms. In particular, we propose a novel adversarial attack framework for both white-box and black-box settings based on the non-convex Frank-Wolfe algorithm. We show in theory that the proposed attack algorithms are efficient with an O(1/ √ T ) convergence rate. The empirical results of attacking Inception V3 model and ResNet V2 model on the ImageNet dataset also verify the efficiency and effectiveness of the proposed algorithms. More specific, our proposed algorithms attain the highest attack success rate in both white-box and black-box attacks among all baselines, and are more time and query efficient than the stateof-the-art.
The paper investigates the Frank-Wolfe (FW) algorithm for constructing adversarial examples both in a white-box and black-box setting. The authors provide both a theoretical analysis (convergence to a stationary point) and experiments for an InceptionV3 network on ImageNet. The main claim is that the proposed algorithm can construct adversarial examples faster than various baselines (PGD, I-FGSM, CW, etc.), and from fewer queries in a black-box setting.
The FW algorithm is a classical method in optimization, but (to the best of my knowledge) has not yet been evaluated yet for constructing adversarial examples. Hence it is a natural question to understand whether FW performs significantly better than current algorithms in this context. Indeed, the authors find that FW is 6x - 20x faster for constructing white-box adversarial examples than a range of relevant baseline, which is a significant speed-up. However, there are several points about the experiments that are unclear to me:
- It is well known that the running times of optimization algorithms are highly dependent on various hyperparameters such as the step size. But the authors do not seem to describe how they chose the hyperparameters for the baselines algorithms. Hence it is unclear how large the running time improvement is compared to a well-tuned baseline.
- Other algorithms in the comparison achieve a better distortion (smaller perturbation). Since finding an adversarial with smaller perturbation is a harder problem, it is unclear how the algorithms compare for finding adversarial examples with similar distortion. Instead of reporting a single time-vs-distortion data point, the authors could show the full trade-off curve.
- The authors only provide running times, not the number of iterations. In principle all the algorithms should have a similar bottleneck in each iteration (computing a gradient for the input image), but it would be good to verify this with an iteration count vs success rate (or distortion) plot. This would also allow the authors to compare their theoretical iteration bound with experimental data.
In addition to these three main points, the authors could strengthen their results by providing experiments on another dataset (e.g., CIFAR-10) or model architecture (e.g., a ResNet), and by averaging over a larger number of test data points (currently 200).
Overall, I find the paper a promising contribution. But until the authors provide a more thorough experimental evaluation, I hesitate to recommend acceptance.
Additional comments:
The introduction contains a few statements that may paint an incomplete or confusing picture of the current literature in adversarial attacks on neural networks:
  
* The abstract claims that the poor time complexity of adversarial attacks limits their practical usefulness. However, the running time of attacks is typically measured in seconds and should not be the limiting element in real-world attacks on deep learning systems. I am not aware of a setting where the running time of an attack is the main computational bottleneck (outside adversarial training).
* The introduction distinguishes between "gradient-based methods" and "optimization-based methods". This distinction is potentially confusing to a reader since the gradient-based methods can be seen as optimization algorithms, and the optimization-based methods rely on gradients.
* The introduction claims that black-box attacks need to estimate gradients coordinate-wise. However, this is not the case already in some of the prior work that uses random directions for estimating gradients (e.g., the cited paper by Ilyas et al.)
I encourage the authors to clarify these points in an updated version of their paper.