Adversarial training is a principled approach for training robust neural networks. Despite of tremendous successes in practice, its theoretical properties still remain largely unexplored. In this paper, we provide new theoretical insights of gradient descent based adversarial training by studying its computational properties, specifically on its implicit bias. We take the binary classification task on linearly separable data as an illustrative example, where the loss asymptotically attains its infimum as the parameter diverges to infinity along certain directions. Specifically, we show that for any fixed iteration T , when the adversarial perturbation during training has proper bounded 2 -norm, the classifier learned by gradient descent based adversarial training converges in direction to the maximum 2 -norm margin classifier at the rate of O(1/ √ T ), significantly faster than the rate O (1/ log T ) of training with clean data. In addition, when the adversarial perturbation during training has bounded q -norm with q ≥ 1, the resulting classifier converges in direction to a maximum mixed-norm margin classifier, which has a natural interpretation of robustness, as being the maximum 2 -norm margin classifier under worst-case q -norm perturbation to the data. Our findings provide theoretical backups for adversarial training that it indeed promotes robustness against adversarial perturbation.
**Contributions:**
This paper extends results on the implicit bias of gradient descent (GD) Soudry et al. (2018)[3] for the case of gradient descent based adversarial training (GDAT).
**1) Theoretical result for L2 norm:** Convergence of standard risk in GDAT is significantly faster than its counterpart in the standard clean training using GD: For any fixed iteration T, when the adversarial perturbation during training has bounded l2-norm, the classifier learned by GDAT converges to the maximum l2-norm margin classifier at the rate of O(1/√T), exponentially faster than the rate O (1/ log T ) obtained when training with only cleaned data Soudry et al. (2018) [3].
**2) theoretical result for Lq norm:** When the adversarial perturbation during training has bounded l_q-norm with q > 1, with a proper choice of c, the gradient descent based adversarial training is directionally convergent to a maximum mixed-norm margin classifier, which has a natural interpretation of robustness, as being the maximum $l_2$-norm margin classifier under worst-case $l_q$-norm perturbation to the data.
The paper is well written and easy to understand. I haven’t checked the proofs, but I focused on readability and motivations. 
I have the following questions.
**Theoretical questions:**
- Can you please explain the intuition of lemma 3.2 and 3.3 for convergence?
**Experiments questions:**
Keeping in mind that the core of the paper is about proving that GDAT enjoys the same implicit bias as GD and better convergence performance on clean data, I can’t help myself by asking you how these properties reflect in experiments with deep Neural Networks:
    - Can you please clarify your motivation for the reduction of the performance gap when the with of the hidden layer increases (Figure 2)? It is not clear what do you mean by ...“as network width increases, the margin on the samples outputted by the hidden layer also increases”...
My personal interpretation of the results is that the inner maximization problem is much harder to solve and being approximated, the effect of the GDAT is much less prominent, meaning that PGD is not enough to benefit from adversarial training in this sense. What do you think about it?
    - Take away message of the paper: adversarial training accelerates convergence. Theoretical results are for the training empirical loss.
Is this also the case of the validation loss? You observe this behavior for the case of a single MLP on MNIST binary problem 2 vs 9 (appendix E) claiming that adversarial training improves generalization performance.
This is not observed in the literature, it is actually the opposite [1, 2]: it exists a tension between robustness and performance on the clean test set. Why do you think you are not observing this? My guesses:
    - you are analysing a very simple binary problem
    - your model is very simple and comparable to a linear classifier, so your theoretical results hold.
[1][1805.12152 Robustness May Be at Odds with Accuracy](https://arxiv.org/abs/1805.12152)
[2][1810.12715 On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://arxiv.org/abs/1810.12715)
[3][1710.10345 The Implicit Bias of Gradient Descent on Separable Data](https://arxiv.org/abs/1710.10345)