Pre-trained language models (PLMs) fail miserably on adversarial attacks. To improve the  robustness, adversarial data augmentation (ADA) has been widely adopted , which attempts to cover more search space of adversarial attacks by adding the adversarial examples during training. However, the number of adversarial examples added by ADA is extremely insufficient due to the enormously large search space. In this work, we propose a simple and effective method to cover  much larger proportion of the attack search space, called Adversarial Data Augmentation with Mixup (MixADA ). Specifically, MixADA linearly interpolates the representations of pairs of training examples to form new virtual samples, which are more abundant and diverse than the discrete adversarial examples used in conventional ADA. Moreover, to  evaluate the robustness of different models fairly , we adopt a challenging  setup, which dynamically generates new adversarial examples for each model. In the text classification experiments of BERT and RoBERTa, MixADA achieves significant robustness gains under two strong adversarial attacks and alleviates the performance degradation of ADA on the original data. Our source codes will be released to support further explorations .
Pretrained language models (PLMs) perform poorly under adversarial attacks. To improve the adversarial robustness, adversarial data augmentation (ADA) has been widely adopted to cover more search space of adversarial attacks by adding textual adversarial examples during training. However, the number of adversarial examples for text augmentation is still extremely insufficient due to the exponentially large attack search space. In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA ). Specifically, AMDA linearly interpolates the representations of pairs of training samples to form new virtual samples, which are more abundant and diverse than the discrete text adversarial examples in conventional ADA. Moreover, to fairly evaluate the robustness of different models , we adopt a challenging evaluation setup, which generates a new set of adversarial examples targeting each model. In text classification experiments of BERT and RoBERTa, AMDA achieves significant robustness gains under two strong adversarial attacks and alleviates the performance degradation of ADA on the clean data. Our code is released at: URL .
R
Pre-trained
Pretrained
R
fail miserably on
perform poorly under
A
adversarial
D
, which attempts
R
the
textual
R
added by ADA is
for text augmentation is still
R
enormously large
exponentially large attack
A
a
R
Data Augmentation with Mixup (MixADA
and Mixup Data Augmentation (AMDA
R
MixADA
AMDA
R
examples
samples
R
adversarial examples used
text adversarial examples
A
fairly
D
fairly
A
evaluation
R
dynamically generates new adversarial examples for
generates a new set of adversarial examples targeting
D
the
R
MixADA
AMDA
R
original
clean
R
source codes will be released to support further explorations
code is released at: URL