Recent work has demonstrated the vulnerability of modern text classifiers to universal adversarial attacks , which are input-agnostic sequences of words added to text processed by classifiers. Despite being successful, the word sequences produced in such attacks are often ungrammatical and can be easily distinguished from natural text. We develop adversarial attacks that appear closer to natural English phrases and yet confuse classification systems when added to benign inputs. We leverage an adversarially regularized autoencoder (ARAE) (Zhao et al., 2018a) to generate triggers and propose a gradient-based search that aims to maximize the downstream classifier's prediction loss. Our attacks effectively reduce model accuracy on classification tasks while being less identifiable than prior models as per automatic detection metrics and human-subject studies. Our aim is to demonstrate that adversarial attacks can be made harder to detect than previously thought and to enable the development of appropriate defenses. 1 1 Introduction Adversarial attacks have recently been quite successful in foiling neural text classifiers (Jia and Liang, 2017; Ebrahimi et al., 2018). Universal adversarial attacks (Wallace et al., 2019; Behjati et al., 2019) are a sub-class of these methods where the same attack perturbation can be applied to any input to the target classifier . These attacks, being input-agnostic, point to more serious shortcomings in trained models since they do not require re-generation for each input. However, the attack sequences generated by these methods are often meaningless and irregular text (e.g., zoning tapping fiennes from Wallace et al. (2019)). While  Equal contribution 1 Our code is available at https://github.com/ Hsuan-Tung/universal_attack_natural_trigger . human readers can easily identify them as unnatural, one can also use simple heuristics to spot such attacks. For instance, the words in the above attack trigger have an average frequency of 14 compared to 6700 for words in benign inputs in the Stanford Sentiment Treebank (SST) (Socher et al., 2013). In this paper, we design natural attack triggers by using an adversarially regularized autoencoder (ARAE) (Zhao et al., 2018a), which consists of an auto-encoder and a generative adversarial network (GAN). We develop a gradient-based search over the noise vector space to identify triggers with a good attack performance. Our method  Natural Universal Trigger Search (NUTS)  uses projected gradient descent with l 2 norm regularization to avoid using out-of-distribution noise vectors and maintain the naturalness of text generated. 2 Our attacks perform quite well on two different classification tasks  sentiment analysis and natural language inference (NLI). For instance, the phrase combined energy efficiency , generated by our approach, results in a classification accuracy of 19.96% on negative examples on the Stanford Sentiment Treebank (Socher et al., 2013). Furthermore, we show that our attack text does better than prior approaches on three different measures  average word frequency, loss under the GPT-2 language model (Radford et al., 2019), and errors identified by two online grammar checking tools (scr; che). A human judgement study shows that up to 77% of raters find our attacks more natural than the baseline and almost 44% of humans find our attack triggers concatenated with benign inputs to be natural. This demonstrates that using techniques similar to ours, adversarial attacks could be made much harder to detect than previously thought and we require the development of appropriate defenses in the long term for securing our NLP models. Input-dependent attacks These attacks generate specific triggers for each different input to a classifier. Jia and Liang (2017) fool reading comprehension systems by adding a single distractor sentence to the input paragraph. Ebrahimi et al. (2018) replace words of benign texts with similar tokens using word embeddings. Similarly, Alzantot et al. (2018) leverage genetic algorithms to design word-replacing attacks. Zhao et al. (2018b) adversarially perturb latent embeddings and use a text generation model to perform attacks. Song et al. (2020) develop natural attacks to cause semantic collisions, i.e. make texts that are semantically unrelated judged as similar by NLP models. Universal attacks Universal attacks are input-agnostic and hence, word-replacing and embedding-perturbing approaches are not applicable. Wallace et al. (2019) and Behjati et al. (2019) concurrently proposed to perform gradient-guided searches over the space of word embeddings to choose attack triggers. In both cases, the attack triggers are meaningless and can be easily detected by a semantic checking process. In contrast, we generate attack triggers that appear more natural and retain semantic meaning. In computer vision, GANs have been used to create universal attacks (Xiao et al., 2018; Poursaeed et al., 2018). Concurrent to our work, Atanasova et al. (2020) design label-consistent natural triggers to attack fact checking models. They first predict unigram triggers and then use a language model conditioned on the unigram to generate natural text as the final attack, while we generate the trigger directly. We build upon the universal adversarial attacks proposed by Wallace et al. (2019). To enable natural attack triggers, we use a generative model which produces text using a continuous vector input, and perform a gradient-guided search over this input space. The resulting trigger, which is added to benign text inputs, is optimized so as to maximally increase the loss under the target classification model. Problem formulation Consider a pre-trained text classifier F to be attacked. Given a set of benign input sequences { x } with the same ground truth label y , the classifier has been trained to predict F ( x ) = y . Our goal is to find a single input-Figure 1: Overview of our attack. Based on the gradient of the target model's loss function, we iteratively update the noise vector n with small perturbation to obtain successful and natural attack triggers. agnostic trigger, t , that when concatenated 3 with any benign input, causes F to perform an incorrect classification, i.e., F ([ t ; x ]) (cid:54) = y , where ; represents concatenation. In addition, we also need to ensure the trigger t is natural fluent text. Attack trigger generation To ensure the trigger is natural, fluent and carries semantic meaning, we use a pre-trained adversarially regularized autoencoder (ARAE) (Zhao et al., 2018a) (details in Section 4). The ARAE consists of an encoder-decoder structure and a GAN (Goodfellow et al., 2014). The input is a standard Gaussian noise vector n , which is first mapped to a latent vector z by the generator. Then the decoder uses this z to generate a sequence of words  in our case, the trigger t . This trigger is then concatenated with a set of benign texts { x } to get full attack texts { x (cid:48) } . The overall process can be formulated as follows: z = GENERATOR ( n ); t = DECODER ( z ); x (cid:48) = [ t ; x ] We then pass each x (cid:48) into the target classifier and compute the gradient of the classifier's loss with respect to the noise vector,  n L ( F ( x (cid:48) ) , y ) . Back-propagating through the decoder is not straightforward since it produces discrete symbols. Hence, we use a reparameterization trick similar to the trick in Gumbel softmax (Jang et al., 2017) to sample words from the output vocabulary of ARAE model as a one-hot encoding of triggers, while allowing gradient backpropagation. Figure 1 provides an overview of our attack algorithm, which we call Natural Universal Trigger Search (NUTS). dard multi-variant Gaussian distribution. While we can change this noise vector to produce different outputs, simple gradient search may veer significantly off-course and lead to bad generations. To prevent this, following Carlini and Wagner (2017), we use projected gradient descent with an l 2 norm constraint to ensure the noise n is always within a limited ball around n 0 . We iteratively update n as: n t +1 =  B (cid:15) ( n 0 ) [ n t +   n t L ( F ( x (cid:48) ) , y )] , (1) where  B (cid:15) ( n 0 ) represents the projection operator with the l 2 norm constraint B (cid:15) ( n 0 ) = { n | (cid:107) n  n 0 (cid:107) 2  (cid:15) } . We try different settings of attack steps, (cid:15) and  , selecting the value based on the quality of output triggers. In our experiments, we use 1000 attack steps with (cid:15) = 10 and  = 1000 . Final trigger selection Since our process is not deterministic, we initialize multiple independent noise vectors (256 in our experiments) and perform our updates (1) to obtain many candidate triggers. Then, we re-rank the triggers to balance both target classifier accuracy m 1 (lower is better) and naturalness in terms of the average per-token cross-entropy under GPT-2, m 2 (lower is better) using the score m 1 + m 2 . We select  = 0 . 05 to balance the difference in scales of m 1 and m 2 . We demonstrate our attack on two tasks  sentiment analysis and natural language inference . We use the method of Wallace et al. (2019) as a baseline 4 and use the same datasets and target classifiers for comparison. For the text generator, we use an ARAE model pre-trained on the 1 Billion Word dataset (Chelba et al., 2014). For both our attack (NUTS) and the baseline, we limit the vocabulary of attack trigger words to the overlap of the classifier and ARAE vocabularies. We generate triggers using the development set of the tasks and report results on test set (results on both sets in Appendix). Defense metrics We employ three simple defense metrics to measure the naturalness of attacks: 1. Word frequency: The average frequency of words in the trigger, computed using empirical estimates from the training set of the target classifier. 4 The baseline attack uses beam search to enlarge the search space in each step. We also tried the baseline attack with 256 random initializations followed by selecting the final trigger using the same criterion as our attack, but its attack success/naturalness remained unchanged. grammar checkers  Scribens (scr) and Chegg (che). 2. Language model loss: The average per-token cross-entropy loss under a pre-trained language model  GPT-2 (Radford et al., 2019). 3. Automatic grammar checkers: We calculate the average number of errors in the attack sequences using two online grammar checkers  Scribens (scr) and Chegg Writing (che). Setup We use a 2-layer LSTM (Hochreiter and Schmidhuber, 1997) followed by a linear layer for sentiment predictions. The model is trained on the binary Stanford Sentiment Treebank (SST) (Socher et al., 2013), using AllenNLP (Gardner et al., 2018). To avoid generating sentiment words in the trigger and directly changing the instance's sentiment, we exclude a list of sentiment words (sen) from the trigger vocabulary, following Wallace et al. (2019). Results Table 2 (top half) captures the results of both our attack and the baseline (Wallace et al., 2019). Our method is able to reduce the classifier's test accuracy significantly, down to 8.55% in the best attack case. Although less successful, our triggers are much more natural, fluent and readable than the baseline. Figure 2 shows the difference in statistics between benign text and each attack according to the metrics of word frequency and GPT-2 loss. Our generated triggers are much closer in these statistics to the original text inputs than the NUTS (our attack) Baseline (Wallace et al., 2019) Task Trigger Test Trigger text Classifier Trigger text Classifier length data accuracy accuracy SST No trigger + -89.00% -89.00% -82.57% -82.57% 8 + the accident forced the empty windows shut down 26.95% collapses soggy timeout energy energy freshness intellect genitals 15.51% will deliver a deeply affected children from parents 8.55% sunny vitality blessed lifetime lifetime counterparts without pitfalls 2.85% SNLI No trigger + -89.76% -89.76% 0 -86.52% -86.52% -79.83% -79.83% 8 + some black women taking the photo last month 0.00% mall destruction alien whatsoever shark pasture picnic no 0.00% 0 the man drowned in hospital and died in 3.26% cats rounds murder pandas in alien spacecraft mars 0.00% they are helping for training achievement for a 26.78% human humans initiate accomplishment energies near objects near 23.02% Table 2: Attack results on SST and SNLI. baseline. Further, as shown in Table 1, two grammar checkers (scr; che) report 12.50% and 21.88% errors per word on our attack triggers, compared to 15.63% and 28.13% for the baseline. Setup We use the SNLI dataset (Bowman et al., 2015) and the Enhanced Sequential Inference Model (ESIM) (Chen et al., 2017) with GloVe embeddings (Pennington et al., 2014) as the classifier. We attack the classifier by adding a trigger to the front of the hypothesis. Results From Table 2, we see that both our attack and the baseline decrease the accuracy to almost 0% on entailment and neutral examples. On contradiction examples, our attack brings the accuracy down to 26 . 78% while the baseline decreases it to 23 . 02% . Although less successful, our attacks are much more natural than the baseline. In Figure 2, our attacks are closer to the word frequency of benign inputs and even achieve a lower GPT-2 loss than the benign text. In Table 1, two grammar checkers (scr; che) also report lower errors on our attacks compared to the baseline. To further validate that our attacks are more natural than baseline, we perform a human-subject study on Amazon Mechanical Turk. We collect ratings by: (1) providing a pair of our trigger vs baseline trigger (with and without benign text) and asking the worker to select the more natural one; (2) providing a piece of text (our attack text/baseline attack text/benign input) and asking the human to determine whether it is naturally generated or not. Both conditions allow the human to choose a Not sure option. We generated attack triggers with lengths of 3, 5, and 8 (see Appendix for details) and created 450 comparison pairs for (1) and 675 pieces of text (225 for each type) for (2). For each instance, we collect 5 different human judgements and report average scores. workers find our attack trigger to be more natural than the baseline while 61 . 16% judge our attack to be more natural even when concatenated with benign text. The other table shows 44 . 27% human subjects think our attack inputs are naturally generated. Although it is lower than the 83 . 11% for real natural inputs, it is still significantly higher than the 22 . 84% of baseline attack inputs, which shows that our attacks are more natural and harder to detect than the baseline for humans. Similar to Wallace et al. (2019), we also evaluate the attack transferability of our universal adversarial attacks to different models and datasets. A transferable attack further decreases the assumptions being made: for instance, the adversary may not need white-box access to a target model and instead generate attack triggers using its own model to attack the target model. We first evaluate transferability of our attack across different model architectures. Besides the LSTM classifier in Section 4.1, we also train a BERT-based classifier on the SST dataset with 92.86% and 91.15% test accuracy on positive and negative data. From Table 4, we can see that the transferred attacks, generated for the LSTM model, lead to 14%  51% accuracy drop on the target BERT model. We also evaluate attack transferability across different datasets. In addition to the SST dataset in Section 4.1, we train a different LSTM classifier with the same model architecture on the IMDB sentiment analysis dataset, which gets 89.75% and 89.85% test accuracy on positive and negative data. Our attacks transfer in this case also, leading to accuracy drops of 18%  34% on the target model (Table 4). We developed universal adversarial attacks with natural triggers for text classification and experimentally demonstrated that our model can generate attack triggers that are both successful and appear natural to humans. Our main goals are to demonstrate that adversarial attacks can be made harder to detect than previously thought and to enable the development of appropriate defenses. Future work can explore better ways to optimally balance attack success and trigger quality, while also investigating ways to detect and defend against them. The techniques developed in this paper have potential for misuse in terms of attacking existing NLP systems with triggers that are hard to identify and/or remove even for humans. However, our intention is not to harm but instead to publicly release such attacks so that better defenses can be developed in the future. This is similar to how hackers expose bugs/vulnerabilities in software publicly. Particularly, we have demonstrated that adversarial attacks can be harder to detect than previously thought (Wallace et al., 2019) and therefore can present a serious threat to current NLP systems. This indicates our work has a long-term benefit to the community. Further, while conducting our research, we used the ACM Ethical Code as a guide to minimize harm. Our attacks are not against real-world machine learning systems. We are grateful to the anonymous reviewers at NAACL for valuable feedback. We thank Austin Wang and Michael Hu for suggestions on using Amazon Mechanical Turk.
abstain abstain objective abstain method objective abstain abstain abstain abstain other abstain abstain method objective method method result result result objective other other other other other other other other other objective other abstain objective abstain method other other other objective abstain other abstain method other other method other method other method method abstain method method method abstain method abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain method abstain other other