We present a simple yet effective T argeted A dversarial T raining ( TAT ) algorithm to improve adversarial training for natural language understanding. The key idea is to introspect current mistakes and prioritize adversarial training steps to where the model errs the most. Experiments show that TAT can significantly improve accuracy over standard adversarial training on GLUE and attain new state-of-the-art zero-shot results on XNLI. Our code will be released at: https://github. com/namisan/mt-dnn . Adversarial training has proven effective in improving model generalization and robustness in computer vision (Madry et al., 2017; Goodfellow et al., 2014) and natural language processing (NLP) (Zhu et al., 2019; Jiang et al., 2019; Cheng et al., 2019; Liu et al., 2020a; Pereira et al., 2020; Cheng et al., 2020). It works by augmenting the input with a small perturbation to steer the current model prediction away from the correct label, thus forcing subsequent training to make the model more robust and generalizable. Aside from some prior work in computer vision (Dong et al., 2018; Tramr et al., 2017), most adversarial training approaches adopt non-targeted attacks, where the model prediction is not driven towards a specific incorrect label. In NLP, the cutting-edge research in adversarial training tends to focus on making adversarial training less expensive (e.g., by reusing backward steps in FreeLB (Zhu et al., 2019)) or regularizing rather than replacing the standard training objective (e.g., in virtual adversarial training (VAT) (Jiang et al., 2019)). By contrast, in this paper, we investigate an orthogonal direction by augmenting adversarial training with introspection capability and adopting targeted attacks to focus on where the model errs the  Equal contribution. most. We observe that in many NLP applications, the error patterns are non-uniform. For example, in the MNLI development set (in-domain), standard fine-tuned BERT model tends to misclassify a non-neutral instance as neutral more often than the opposite label (Figure 1 top). We thus propose Targeted Adversarial Training (TAT), a simple yet effective algorithm for adversarial training. For each instance, instead of taking adversarial steps away from the gold label, TAT samples an incorrect label proportional to how often the current model makes the same error in general, and takes adversarial steps towards the chosen incorrect label. To our knowledge, this is the first attempt to apply targeted adversarial training to NLP tasks. In our experiments, this leads to significant improvement over standard non-adversarial and adversarial training alike. For example, in the MNLI development set, TAT produced an accuracy gain of 1.7 absolute points (Figure 1 bottom). On the overall GLUE benchmark, TAT outperforms state-of-the-art non-targeted adversarial training methods such as FreeLB and VAT, and enables the BERTBASE model to perform comparably to the BERTLARGE model with standard training. The benefit of TAT is particularly pronounced in out-domain settings, such as in zero-shot learning in natural language inference, attaining new state-of-the-art cross-lingual results on XNLI. In this paper, we focus on fine-tuning BERT models (Devlin et al., 2018) in our investigation of targeted adversarial training, as this approach has proven very effective for a wide range of NLP tasks. The training algorithm seeks to learn a function f ( x ;  ) : x  C as parametrized by  , where C is the class label set. Given a training dataset D of input-output pairs ( x, y ) and the loss function l ( ., . ) (e.g., cross entropy), the standard training objective would minimize the empirical risk: min  E ( x,y )  D [ l ( f ( x ;  ) , y )] . By contrast, in adversarial training, as pioneered in computer vision (Goodfellow et al., 2014; Hsieh et al., 2019; Madry et al., 2017; Jin et al., 2019), the input would be augmented with a small perturbation that maximize the adversarial loss: min  E ( x,y )  D [max  l ( f ( x +  ;  ) , y )] , where the inner maximization can be solved by projected gradient descent (Madry et al., 2017). Recently, adversarial training has been successfully applied to NLP as well (Zhu et al., 2019; Jiang et al., 2019; Pereira et al., 2020). In particular, FreeLB (Zhu et al., 2019) leverages the free adversarial training idea (Shafahi et al., 2019) by reusing the backward pass in gradient computation to carry out inner ascent and outer descent steps simultaneously. SMART (Jiang et al., 2019) instead Algorithm 1 TAT Input: T : the total number of iterations, X = { ( x 1 , y 1 ) , ..., ( x n , y n ) } : the dataset, f ( x ;  ) : the machine learning model parametrized by  ,  2 : the variance of the random initialization of perturbation  , (cid:15) : perturbation bound, K : the number of iterations for perturbation estimation,  : the step size for updating perturbation,  : the global learning rate,  : the smoothing proportion of adversarial training in the augmented learning objective,  : the projection operation and C : the classes. 1: for t = 1 , .., T do 2: for ( x, y )  X do 3:   N (0 ,  2 I ) 4: y t = sample ( C \ y ) 5: for m = 1 , .., K do 6: g adv    l ( f ( x +  ;  ) , y t ) 7:    (cid:107)  (cid:107)   (cid:15) (   g adv ) 8: end for 9: g     l ( f ( x ;  ) , y ) +    l ( f ( x ;  ) , f ( x +  ;  )) 10:     g  11: end for 12: end for Output:  regularizes the standard training objective using virtual adversarial training (Miyato et al., 2018): min  E ( x,y )  D [ l ( f ( x ;  ) , y )+  max  l ( f ( x +  ;  ) , f ( x ;  ))] (1) Effectively, the adversarial term encourages smoothness in the input neighborhood, and  is a hyperparameter that controls the trade-off between standard errors and adversarial errors. In standard adversarial training, the algorithm simply tries to perturb the input x away from the gold label y given the current parameters  . It is agnostic to which incorrect label f ( x ) might be steered towards. By contrast, in Targeted Adversarial Training (TAT), we would explicitly pick a target y t (cid:54) = y and try to steer the model towards y t . Intuitively, we would like to focus training on where the model currently errs the most. We accomplish this by keeping a running tally of e ( y, y t ) , which is the current expected error of predicting y t when the gold label is y , and sample y t from C \ y = C  { y } in proportion to e ( y, y t ) . See Algorithm 1 for details. TAT can be applied to the Methods MNLI-m/mm QQP RTE QNLI MRPC CoLA SST STS-B Average Acc Acc/F1 Acc Acc Acc/F1 Mcc Acc P/S Corr Score Standard ( BERTLARGE ) dev 86.3/86.2 91.3/88.4 71.1 92.4 85.8/89.5 61.8 93.5 89.6/89.3 84.0 Standard ( BERTLARGE ) test 86.7/85.9 72.1/89.3 70.1 92.7 85.4/89.3 60.5 94.9 87.6/86.5 82.4 Standard dev 84.5/84.4 90.9/88.3 63.5 91.1 84.1/89.0 54.7 92.9 89.2/88.8 81.5 FreeLB dev 85.4/85.5 91.4/88.4 70.4 91.5 86.2/90.3 59.1 93.2 89.7/89.1 83.5 VAT dev 85.5/85.7 91.5/88.5 71.2 91.7 87.7/91.3 58.2 93.3 90.0/89.4 83.7 TAT dev 86.2/85.9 91.8/89.1 72.6 92.2 88.2/91.5 58.5 93.6 90.8/89.6 84.2 Standard test 84.6/83.4 71.2/89.2 66.4 90.5 84.8/88.9 52.1 93.5 87.1/85.8 80.0 TAT test 85.8/84.8 72.8/89.6 69.7 92.4 88.2/91.1 59.8 94.5 89.7/89.0 82.8 Table 1: Comparison of standard and adversarial training methods on GLUE. original adversarial training or virtual adversarial training alike. In this paper, we focus on adapting virtual adversarial training (VAT) (Jiang et al., 2019). The two lines in blue color are the only change from VAT. We initialize e ( y, y t ) with uniform distribution and update them in each epoch. We conducted an oracle experiment where e ( y, y t ) was taken from the confusion matrix from standard training and found that it performed similarly as our online version. It is more challenging to apply TAT to regression tasks, as we would need to keep track of a continuous error distribution. To address this problem, we quantize the value range into ten bins and apply TAT similarly as in the classification setting (once a bin is chosen, a value is sampled uniformly within). We compare targeted adversarial training (TAT) with standard training and state-of-the-art adversarial training methods such as FreeLB (Zhu et al., 2019) and VAT (Miyato et al., 2018; Jiang et al., 2019). We use the standard uncased BERTBASE model (Devlin et al., 2018), unless noted otherwise. Due to the additional overhead incurred during training, adversarial methods are somewhat slower than standard training. Like VAT, TAT requires an additional K adversarial steps compared to standard training. In practice, K = 1 suffices for TAT and VAT, so they are just slightly slower (roughly 2 times compared to standard training). FreeLB, by contrast, typically requires 2-5 steps to attain good performance, so is significantly slower. Our implementation is based on the MT-DNN toolkit (Liu et al., 2020b). We follow the default hyperparameters used for fine-tuning the uncased BERT base model (Devlin et al., 2018; Liu et al., 2020b). Specifically, we use 0 . 1 for the dropout rate except 0.2 for MNLI, 0 . 01 for the weight de-cay rate and the Adamax (Kingma and Ba, 2014) optimizer with the default Lookahead (Zhang et al., 2019) to stabilize training. We select the learning rate from { 5e  5 , 1e  4 } for all the models. The maximum training epoch is set to 6, and the we follow (Jiang et al., 2019) to set adversarial training hyperparameters: (cid:15) = 1e  5 and  = 1e  4 . In our experiments, we simply set  = 1 in Eq 1. We first compare adversarial training methods on the standard GLUE benchmark (Wang et al., 2018). See Table 1 for the results 1 . TAT consistently outperforms both standard training and the state-of-the-art adversarial training methods of FreeLB and VAT. Remarkably, BERTBASE with targeted adversarial training performs on par with BERTLARGE with standard training overall, and outperforms the latter by a large margin on tasks with smaller datasets such as RTE, MRPC and STS-B, which illustrates the benefit of TAT in improving model generalizability. 1 Due to restriction on the number of submissions by the GLUE organizers, we only compared TAT with the published results from (Devlin et al., 2018) on the test set. Next, we compare standard and adversarial training in generalizability to out-domain datasets. Specifi-cally, we fine-tune BERTBASE on the MNLI training data and evaluate it on various natural language inference test sets: HANS (McCoy et al., 2019), SNLI (Bowman et al., 2015), SciTail (Khot et al., 2018), MeNLI (Romanov and Shivade, 2018). See Table 2 for the results. TAT substantially outperforms standard training and state-of-the-art adversarial training methods. Interestingly, the gains are particularly pronounced on the two hardest datasets, HANS and MedNLI. HANS used heuristic rules to identify easy instances for MNLI-trained BERT models and introduced modifications to make them harder. MedNLI is from the biomedical domain, which is substantially different from the general do-main of MNLI. This provides additional evidence that targeted adversarial training is especially effective in enhancing generalizability in out domains. We also conducted zero-shot evaluation in the cross-lingual setting by comparing standard and adversarial training on XNLI (Conneau et al., 2018). Specifically, a cross-lingual language model is fine-tuned using the English NLI dataset and then tested on datasets of other languages. Following Conneau et al. (2019), we used the pre-trained XLM-R large model in our experiments, and compare targeted adversarial training (XLM-R+TAT) with state-of-the-art systems that use standard training (XLM-R) and adversarial training (XLM-R+R3F/R4F) (Aghajanyan et al., 2020), as well as another state-of-the-art language model InfoXLM (Chi et al., 2020). To ensure fair comparison, we also report the results from our reimplementation of XLM-R (Conneau et al., 2018) (XLM-R Reprod ). See Table 3 for the results. Targeted adversarial training (TAT) demonstrates a clear advantage in improving zero-shot transfer learning across languages, especially for languages most different from English, such as Urdu. Overall, TAT produces a new state-of-the-art result of 81.7% over 15 languages on XNLI. As we have seen in Figure 1 earlier, TAT reduces the errors across the board on MNLI development set. To understand how TAT improves performance, we conducted a more detailed analysis by subdividing the dataset based on the degree of human agreement. Here, there are three label classes and each sample instance has 5 human annotations. The samples can be divided into four categories: 5-0-0, 4-1-0, 3-2-0, 3-1-1. E.g., 3-1-1 signifies that there are three votes for one label and one for each of the other two labels. In Figure 2, we see that TAT outperforms the baseline consistently over all categories, with higher improvement on the more ambiguous samples, especially for out-domain samples. This suggests that TAT is most helpful for the challenging instances that exhibit higher ambiguity and are more different from training examples. Model en fr es de el bg ru tr ar vi th zh hi sw ur Avg. XLM-R 89.1 84.1 85.1 83.9 82.9 84.0 81.2 79.6 79.8 80.8 78.1 80.2 76.9 73.9 73.8 80.9 XLM-R Reprod 88.1 83.6 84.1 83.0 82.6 83.8 81.7 80.7 80.4 80.7 78.9 80.1 77.8 74.2 74.0 80.9 XLM-R+R3F 89.4 84.2 85.1 83.7 83.6 84.6 82.3 80.7 80.6 81.1 79.4 80.1 77.3 72.6 74.2 81.2 XLM-R+R4F 89.6 84.7 85.2 84.2 83.6 84.6 82.5 80.3 80.5 80.9 79.2 80.6 78.2 72.7 73.9 81.4 InfoXLM 89.7 84.5 85.5 84.1 83.4 84.2 81.3 80.9 80.4 80.8 78.9 80.9 77.9 74.8 73.7 81.4 XLM-R+TAT 89.3 84.2 85.7 83.9 83.7 85.0 82.1 81.0 80.7 81.3 79.7 81.0 78.4 74.1 75.1 81.7 Table 3: Comparison of targeted adversarial training (TAT) and prior state of the art in zero-shot cross-lingual learning on the XNLI test set. has a wider and flatter loss surface, which generally indicates better generalization (Hochreiter and Schmidhuber, 1997; Hao et al., 2019; Li et al., 2018). We present the first study to apply targeted attacks in adversarial training for natural language understanding. understanding. Our TAT algorithm is simple yet effective in improving model generalizability for various NLP tasks, especially in zero-shot learning and for out-domain data. Future directions include: applying TAT in pretraining and other NLP tasks e.g., sequence labeling, exploring alternative approaches for target sampling. We thank Microsoft Research Technology Engineering team for setting up GPU machines. We also thank the anonymous reviewers for valuable discussions.
result abstain abstain other abstain other abstain abstain abstain objective abstain result abstain objective abstain objective result abstain abstain abstain method abstain abstain other abstain abstain other abstain abstain other abstain abstain abstain method method abstain other abstain method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain other abstain objective abstain result abstain other other