Multilingual models have demonstrated impressive cross-lingual transfer performance. However, test sets like XNLI are monolingual at the example level. In multilingual communities, it is common for polyglots to code-mix when conversing with each other. Inspired by this phenomenon, we present two strong black-box adversarial attacks (one word-level, one phrase-level) for multilingual models that push their ability to handle code-mixed sentences to the limit. The former uses bilingual dictionaries to propose perturbations and translations of the clean example for sense disambiguation. The latter directly aligns the clean example with its translations before extracting phrases as perturbations. Our phrase-level attack has a success rate of 89.75% against XLM-R large , bringing its average accuracy of 79.85 down to 8.18 on XNLI. Finally, we propose an efficient adversarial training scheme that trains in the same number of steps as the original model and show that it improves model accuracy. 1 1 Introduction The past year has seen incredible breakthroughs in cross-lingual generalization with the advent of massive multilingual models that aim to learn universal language representations (Pires et al., 2019; Wu and Dredze, 2019; Conneau et al., 2020b). These models have demonstrated impressive cross-lingual transfer abilities: simply fine-tuning them on task data from a high resource language such as English after pretraining on monolingual corpora was sufficient to manifest such abilities. This was observed even for languages with different scripts and no vocabulary overlap (K et al., 2020). However, transferring from one language to another is insufficient for NLP systems to understand multilingual speakers in an increasingly multilingual world (Aronin and Singleton, 2008). In many 1 Code: github.com/salesforce/adversarial-polyglots (a) Aligned words across sentences (b) Extracted candidate perturbations (c) Final multilingual adversary Figure 1: BUMBLEBEE 's three key stages of adversary generation: multilingual societies (e.g., Singapore, Papua New Guinea, etc.), it is common for multilingual interlocutors to produce sentences by mixing words, phrases, and even grammatical structures from the languages in their repertoires (Matras and Sakel, 2007). This is known as code-mixing (Poplack et al., 1988), a phenomenon common in casual conversational environments such as social media and text messages. 2 Hence, it is crucial for NLP systems serving multilingual communities to be robust to code-mixing if they are to understand and establish rapport with their users (Tay, 1989; Bawa et al., 2020) or defend against adversarial polyglots. Although gold standard data (Bali et al., 2014; Patwa et al., 2020) is important for definitively evaluating code-mixed text processing ability, such datasets are expensive to collect and annotate. The 2 Examples of real code-mixing in Appendix A. dizzying range of potential language combinations further compounds the immensity of such an effort. We posit that performance on appropriately crafted adversaries could act as a lower bound of a model's ability to generalize to the distribution simulated by said adversaries, an idea akin to worst-case analysis (Divekar, 1984). For example, Tan et al. (2020b) showed that an NLP system that was robust to morphological adversaries was less perplexed by dialectal text exhibiting morphological variation. Likewise, if a system is robust to code-mixed adversaries constructed from some set of languages, it is reasonable to expect it to also perform better on real code-mixed text in those languages. While they may not fully model the intricacies of real code-mixing (Sridhar and Sridhar, 1980), we believe that they can be useful in the absence of appropriate evaluation data. Hence, we:  Propose two strong black-box adversarial attacks targeting the cross-lingual generalization ability of massive multilingual representations (Fig. 1), demonstrating their effectiveness on state-of-the-art models for natural language inference and question answering. To our knowledge, these are the first two multilingual adversarial attacks. Propose an efficient adversarial training scheme that takes the same number of steps as standard supervised training and show that it creates more language-invariant representations, improving accuracy in the absence of lexical overlap. Multilingual classifiers. Low resource languages often lack support due to the high cost of annotating data for supervised learning. An approach to tackle this challenge is to build cross-lingual representations that only need to be trained on task data from a high resource language to perform well on another under-resourced language (Klementiev et al., 2012). Artetxe and Schwenk (2019) presented the first general purpose multilingual representation using a BiLSTM encoder. Following the success of Transformer models (Vaswani et al., 2017), recent multilingual models like mBERT (De-vlin et al., 2019), Unicoder (Huang et al., 2019), and XLM-R (Conneau et al., 2020a) take the pretraining  fine-tuning paradigm into the multilingual realm by pretraining Transformer encoders on unlabeled monolingual corpora with various language modeling objectives before fine-tuning them on task data from a high-resource language such as English. Code-mixed text processing. Previous research on code-mixed text processing focused on constructing formal grammars (Joshi, 1982) and token-level language identification (Bali et al., 2014; Solorio et al., 2014; Barman et al., 2014), before progressing to named entity recognition and part-of-speech tagging (Ball and Garrette, 2018; AlGhamdi and Diab, 2019; Aguilar and Solorio, 2020). Recent work explores code-mixing in higher-level tasks such as question answering and task-oriented dialogue (Chandu et al., 2019; Ahn et al., 2020). Muller et al. (2020) demonstrate mBERT's ability to transfer to an unseen dialect by exploiting its speakers' tendency to code-mix. A key challenge of developing models that are robust to code-mixing is the availability of code-mixed datasets. Hence, Winata et al. (2019) use a pointer-generator network to generate synthetically code-mixed sentences while Pratapa et al. (2018) explore the use of parse trees for the same purpose. Yang et al. (2020) propose to improve machine translation with code-switching pretraining, replacing words with their translations in a similar manner to masked language modeling (Devlin et al., 2019). These word pairs are constructed from monolingual corpora using cosine similarity. Sitaram et al. (2019) provide a comprehensive survey of code-mixed language processing. Word-level adversaries. Modified inputs aimed at disrupting a model's predictions are known as adversarial examples (Szegedy et al., 2014). In NLP, perturbations can be applied at the character, subword, word, phrase, or sentence levels. Early word-level adversarial attacks (Ebrahimi et al., 2018; Blohm et al., 2018) made use of the target model's gradients to flip individual words to trick the model into making the wrong prediction. However, while the perturbations were adversarial for the target model, perturbed word's original semantics was often not preserved. This could result in the expected prediction changing and making the model appear more brittle than it actually is. Later research addressed this by searching for adversarial rules (Ribeiro et al., 2018) or by constraining the candidate perturbations to the k nearest neighbors in the embedding space (Alzantot et al., 2018; Michel et al., 2019; Ren et al., 2019; Zhang et al., 2019; Li et al., 2019; Jin et al., 2020). Zang Original P: The girl that can help me is all the way across town. et al. (2020) take another approach by making use of a annotated sememes to disambiguate polysemous words, while Tan et al. (2020a) perturb only the words' morphology and encourage semantic preservation via a part-of-speech constraint. Other approaches make use of language models to generate candidate perturbations (Garg and Ramakrish-nan, 2020; Han et al., 2020). Wallace et al. (2019) find phrases that act as universally adversarial perturbations when prepended to clean inputs. Zhang et al. (2020) provide a comprehensive survey. Summary. Existing work on pretrained multilingual models has highlighted their impressive zero-shot cross-lingual transfer ability, though some analyses (K et al., 2020) indicate this could be a result of exploiting lexical overlaps rather than an indication of true cross-lingual understanding. Although language-agnosticity is commonly measured via cross-lingual retrieval tasks such as LAReQA (Roy et al., 2020) and similarity search (Artetxe and Schwenk, 2019), we offer a different perspective in this paper by operationalizing it as a model's ability to handle code-mixing. Existing evaluations for code-mixed text processing focus on gold annotated data, but such datasets are (relatively) expensive to compile and face similar scarcity challenges as those for low-resource languages. Existing word-/phrase-level adversarial attacks probing the limits of model robustness have largely focused on monolingual (English) inputs. In contrast, our adversarial attacks are designed to test the robustness of multilingual models to adversarial code-mixers. Finally, we propose an efficient adversarial training scheme to improve the robustness of said models to code-mixed adversaries. Code-mixing is a phenomenon where a multilingual speaker mixes words, and even grammatical rules, from different languages in a single sentence. This is distinguished from code-switching, which occurs at the inter-sentential level (Kachru, 1978). Extreme code-mixing. Inspired by the proliferation of real-life code-mixing and polyglots, we propose POLYGLOSS and BUMBLEBEE , two multilingual adversarial attacks that adopt the persona of an adversarial code-mixer. We focus on the lexical component of code-mixing, where some words in a sentence are substituted with their equivalents from another language in the interlocutor's repertoire. Borrowed words fall into two categories, nonce borrowing and loanwords, though distinguishing between them is beyond the scope of this work. Since most code-mixers are bilinguals, natural code-mixed sentences tend to be constructed from two languages, with one language determining the syntax of the overall sentence (Poplack et al., 1988). However, in a world with an increasing number of multilingual societies, it is conceivable for code-mixing to occur between more than two languages (Tan, 1988). We take this idea to the extreme to test multilingual representations for their robustness to such cross-lingual lexical variation. Problem formulation. Given a target multilingual model M , a clean example x with the label y , and a set of embedded languages L from which to borrow words, we aim to generate the adversarial example x (cid:48) that maximizes M 's loss. Formally, x (cid:48) = arg max x c  XL ( y, M ( x c )) , (1) where x c  X is a candidate adversary generated by perturbing x , M is a task-specific neural model, and L (  ) is the model's loss function. To obtain a code-mixed adversary, we first generate candidate adversaries by substituting words in the clean example with their equivalents from another language. These substitutions/perturbations can be generated by via machine translation or mined from bilingual dictionaries. Following Myers-Scotton (1997), we will refer to the original example's language as the matrix language and the perturbation's language as the embedded language. Next, we perform beam search on the candidates to find the adversary that maximizes the target model's loss in a black-box manner (Alg. 2 in Appendix B.1). In our implementation, we also keep track of successful adversaries and return the ones with the highest and lowest losses. The former is a stronger adversary, while the latter often has fewer perturbations. More details are in Appendix B.1. Orthographic preservation. When the embedded language uses a different script from the matrix language, code-mixers tend to transliterate borrowed words into the same script (Abuhakema, 2013; Bali et al., 2014). This still poses a significant challenge to multilingual models (Khanuja et al., 2020). We generally preserve the embedded language's script where possible to avoid unfairly penalizing the target model since there is often no standard way of transliterating words. Scalable sense disambiguation. Due to the polysemous nature of many words, translating the right sense is crucial to preserving the word's (and sen-tence's) semantics. Common word sense disambiguation methods (Agirre and Edmonds, 2007) use a sense tagger trained on an annotated sense inventory such as WordNet (Miller, 1995). However, this approach requires individual taggers and sense inventories for each matrix and embedded language, making it a serious challenge to extend POLYGLOSS to low-resource languages. Instead, we propose to filter candidate perturbations using the embedded language translation of the clean example. This is easily done by checking if the candidate perturbation exists in the translation. Since our examples tend to be single sentences, the probability of different senses of the same word occurring in a single sentence is generally low (Conneau et al., 2018; Popel et al., 2020). This approach only requires a machine translation (MT) system and no extra linguistic information, making it highly scalable as long as a supervised (or unsupervised) machine translation system is available. By using gold translations instead of machine translations, it is even possible to mostly Algorithm 1 BUMBLEBEE Require: Clean example-label pair ( x, y ) , Target Model M , Embedded languages L Ensure: Adversarial example x (cid:48) T  TRANSLATE ( x, target-languages = L ) L x  GETLOSS ( M , x, y ) B  { ( L x , x, 0) } (cid:46) Initialize beam P  ALIGNANDEXTRACTPHRASES ( x, T ) while NOTEMPTY ( B ) do L x c , x c , i  POLL ( B ) C  GETCANDIDATES ( x c , P [ i ]) L  GETLOSS ( M , C, y ) (cid:46) Losses for C i  i + 1 UPDATEBEAM ( B, L , C, i ) end while x (cid:48)  POLL ( B ) return x (cid:48) guarantee semantic preservation at the word-level. Although using bilingual dictionaries with our filtering method ensures that the semantics of a borrowed word matches the original, the dictionary's comprehensiveness determines the presence of sufficient candidate adversaries. In addition, POLYGLOSS swaps words at the word level, which may hurt the naturalness of the resulting sentence since it is more common for code-mixers to borrow phrases than individual words (Abuhakema, 2013). A solution to these issues is to replace phrases in the matrix sentence with their equivalents from the reference translations instead of using a dictionary lookup (Alg. 1). A key advantage of this approach is its flexibility and scalability to more languages since it only requires parallel bitexts from the matrix and embedded languages. With the advent of neural sequence-to-sequence models, such bitexts can be easily generated using publicly available MT models. However, a key challenge for this approach is extracting the matrix-embedded phrase pairs from the clean example and its translation. We follow common phrase-based machine translation methods and accomplish this by aligning the matrix and embedded sentences (Koehn, 2010). Implementation details can be found in Appendix B.2. Syntactic preservation. To improve the adver-saries' naturalness, we impose an equivalence constraint (Poplack, 1980), preventing a perturbation from being applied if it is from the same language as the previous word and will disrupt the syntax of the current phrase if applied (Winata et al., 2019). Such disruptions usually occur when borrowing words from languages with a different word order. We first evaluate POLYGLOSS and BUMBLEBEE on XNLI (Conneau et al., 2018), then evaluate the stronger attack on XQuAD (Artetxe et al., 2020). XNLI is a multilingual dataset for natural language inference (NLI) with parallel translations for each example in fifteen languages. Each example comprises a premise, hypothesis, and a label with three possible classes: { contradiction, neutral, entailment } . We construct two more datasets from XNLI: XNLI-13 and XNLI-32. XNLI-13 comprises all XNLI languages except Swahili and Urdu due to the lack of suitable dictionaries for POLYGLOSS . We then translate the English test set into eighteen other languages with MT systems to form XNLI-31, increasing the number of embedded languages POLYGLOSS can draw from. XQuAD is a multilingual dataset for extractive question answering (QA) with parallel translations in eleven languages. In the cross-lingual transfer setting, the models are trained on English data, MNLI (Williams et al., 2018) and SQuAD 1.1 (Rajpurkar et al., 2016), and tested on mulitlingual data, XNLI and XQuAD, respectively. We perturb the premise and hypothesis for NLI and only the question for QA. More experimental details can be found in Appendix D. Matrix language. Although our attacks work with any language as the matrix language, we use English as the matrix language in our experiments Model Clean BUMBLEBEEXLM-R large 75.64 / 61.39 35.32 / 22.52 XLM-R base 68.90 / 53.50 17.95 / 10.33 mBERT base 64.66 / 49.47 20.66 / 11.68 Table 4: BUMBLEBEE results on XQuAD (F 1 /EM). due to the availability of English  T translation models and the prevalence of English as the matrix language in many code-mixing societies. Models. We conduct our experiments on three state-of-the-art massive multilingual encoder models: XLM-RoBERTa, mBERT, and Unicoder, each pretrained on more than 100 languages. From Tables 2 and 3, we observe that all the models are significantly challenged by adversarial code-mixing, though XLM-R large is the most robust to both attacks, likely due to having more parameters. However, even after filtering POLYGLOSS 's candidate perturbations by the gold translations in XNLI-13, we observe an average drop in accuracy of 80.01%, relative to the models' accuracy on the clean XNLI-13. BUMBLEBEE induces even greater performance drops (average relative decrease of 90.96% on XNLI-13), likely due to its word aligner yielding more candidates than POLYGLOSS 's dictionary lookup. Increasing the number of embedded languages POLYGLOSS can draw upon results in greater drops in model performance (average relative decrease in accuracy of 93.66% on XNLI-31). BERTvs. XLM-based. We notice that mBERT is more sensitive to intra-phrasal syntactic disruption than the XLM-based models. mBERT is the most robust to BUMBLEBEE out of all the base models when the equivalence constraint is in place, yet is the least robust to POLYGLOSS . However, the latter trend is replicated for BUMBLEBEE if we remove this constraint (Table 16 in Appendix G). A possible explanation is that XLM-R and Unicoder were trained on monolingual CommonCrawl (CC) data, while mBERT was trained on multilingual Wikipedia, which could be considered as aligned at the article level since there are articles on the same topic in different languages. Hence, it is possible that this helped to align the languages more accurately in the feature space but made it more sensitive to syntactic disruptions. However, many other hyperparameters differ between the two that could have also influenced their robustness. Hence, we leave a rigorous study of these factors to future work. The higher performance of the XLM-based models on clean data can likely be attributed to the CC corpus being an order of magnitude larger than multilingual Wikipedia (Lauscher et al., 2020). Candidate filtering. In the unfiltered setting, it is impossible for POLYGLOSS to discriminate between valid and invalid senses for a given context. Hence, a potential criticism is that the large difference in POLYGLOSS 's success rate between the filtered and unfiltered settings could be attributed to the inappropriate senses of polysemous words being chosen and disrupting the semantics of the sentence. On the other hand, filtering perturbations with reference translations of the sentence shrinks the space of perturbations to  1 per language. Due to the dictionaries' non-exhaustive nature, not every word in the matrix sentence has an entry in the dictionary to begin with, making this filtering step a significant reduction of the space of candidates. To determine the likely cause of the accuracy difference between the filtered and unfiltered settings in XNLI-13, we increase the number of languages available to POLYGLOSS to thirty-one. If the difference between the filtered and unfiltered settings were not due to a lack of sufficient candidates, we should observe only a minor difference between the filtered settings for both XNLI-13 and -31. However, we observe a 69% drop for XLM-R large , indicating that the former accuracy difference is likely due to the reduced number of valid candidates. Phrase-level adversaries. In addition to generating more fluent sentences (Table 1), extracting the candidate perturbations directly from the translations does away with the need for sense disambiguation and increases the number of perturbations per example since it is not limited to a static dictionary. The increased effectiveness of BUMBLEBEE compared to POLYGLOSS (1.13x) is further evidence that a key factor to the success of such adversarial attacks is the availability of sufficient candidates; increasing the dimensionality of the search space increases the probability that an adversarial example for the model exists (Goodfellow et al., 2015). We also include a non-adversarial baseline (Rand.) by sampling candidates from a uniform distribution instead of searching for the worst-case perturbations. Our results in Table 3 indicate that the worst-case performance of multilingual models on code-mixed data may be much lower than the scores reported on human-produced test sets since they were not Model Devanagari Transliterated (Latin) XLM-R large 61.35 41.97 XLM-R base 48.62 30.01 mBERT base 37.70 23.41 Unicoder base 49.34 30.00 Table 5: BUMBLEBEE results on XNLI en,hi using both Devanagari and Latin scripts. created in a targeted, adversarial fashion. Experiments on beam width and a proof of concept for fully unsupervised adversaries are in Appendix E. Transliteration. Since real-life code-mixers often use a single script for the entire sentence, we now test the effect of transliteration on BUMBLEBEE 's success rate for the English + Hindi language pair. We accomplish this by transliterating all candidates from Devanagari into Latin using the dictionaries released by Roark et al. (2020). From Table 5, we see that transliteration significantly affects the robustness of all models, even the XLM-based ones which were pretrained on similar data. XQuAD. We observe that both XLM-R and mBERT are significantly challenged by BUMBLEBEE even though only the question was modified (Table 4). We did not experiment on Unicoder to reduce carbon costs since its performance was almost identical to XLM-R base in our XNLI experiments. POLYGLOSS or BUMBLEBEE ? As expected, inspection of individual adversarial examples revealed that BUMBLEBEE generated more natural sentences than POLYGLOSS since the languages used within phrases were more consistent (Table 1). However, incorrect alignments due to the word aligner's probabilistic nature could introduce occasional noise into the adversarial examples. For example, we found the (en) to be often aligned with    (zh) even though the former is an article and the latter a possessive. We observe that the aligner performs better when the sentences have similar word orders (e.g., English-French vs. English-Chinese) and we can expect the adversaries generated in these settings to be more natural. Hence, we recommend POLYGLOSS when greater preservation of word-level semantics is desired, and BUMBLEBEE when phrase-level perturbations are desired or bilingual dictionaries are unavailable. Discussion. K et al. (2020) noted significant performance drops in XNLI accuracy for mBERT when the premise and hypothesis were in different languages (Fake English vs. { Hindi, Russian, Spanish } ), theorizing this to be an effect of disrupting the model's reliance on lexical overlap. Our experiments in 4 and 5 lend support to this hypothesis. In Table 1, we see multiple examples where the prediction was flipped from contradic-tion to entailment simply by perturbing a few words. If the models did not rely on lexical overlap but performed comparisons at the semantic level, such perturbations should not have severely impacted their performance. Our results on QA also corroborate Lee et al. (2019)'s finding that models trained on SQuAD-style datasets exploit lexical overlap between the question and context. Finally, we propose code-mixed adversarial training (CAT), an extension of the standard adversarial training paradigm (Goodfellow et al., 2015), to improve the robustness of multilingual models to adversarial polyglots. In standard adversarial training, adversarial attacks are run on the training set to generate adversaries for training. However, this makes adversarial training computationally expensive. Hence, we take inspiration from Tan et al. (2020a)'s method of randomly sampling perturbations from an adversarial distribution and generate code-mixed perturbations using word alignment. To generate the code-mixed adversarial training set X (cid:48) , we first compute the adversarial distribution P adv by enumerating the perturbations per embedded language in all successful adversaries (4). Formally, P adv = { f i } i =1 ... | L | , where f i = l i (cid:80) | L | j =1 l j and L is the set of embedded languages. Next, for each clean example x , we sample n languages from P adv before translating the example into the n languages and aligning the translations with x . For sentence-pair classification tasks like NLI, we use a per-sentence n to further increase variation. Intuitively, limiting n improves the example's naturalness and the algorithm's efficiency (the alignment is the most costly step). We then extract phrases from the aligned sentences, yielding our candidate perturbations P . Next, we sample a perturbation with probability  from P for each phrase in x . Reducing  yields more natural sentences since they will be less perturbed. Finally, we apply these perturbations to x , obtaining a CAT example x (cid:48) . Doing this k times for all x in X and adding the result to X yields X (cid:48) (Alg. 3 in Appendix G). In contrast to running the adversarial attack on the training set, sampling perturbations from a distribution does not guarantee that the resulting example will be adversarial to the model. This issue can be mitigated by increasing the number of CAT examples observed during training. However, this would increase the computational cost if we were to train the model for the same number of epochs. Hence, we set k to one less than the number of epochs XLM-R base was fine-tuned for in 4 and train the model for one epoch on the adversarial training set. This exposes the model to more variation in the same number of training steps. Setting. We conduct our experiments on NLI with XLM-R base with no loss of generality. In 4, the model was trained for ten epochs. Hence, we set k = 9 , n = 2 ,  = 0 . 5 for CAT and train all models for a similar number of steps (60k) with the same hyperparameters as 4. We first test the models on the BUMBLEBEE adversaries generated in 4 before directly attacking the model. Next, we construct more realistic settings by running BUMBLEBEE with only 1-2 embedded languages from standard XNLI, Swahili (sw), Hindi (hi), and Urdu (ur). These languages were the lowest resourced in the pretraining data (Conneau et al., 2020a). We also construct another non-adversarial test set from XNLI by randomly choosing hypotheses and premises from different languages (K et al., 2020). Since the original examples are individually monolingual, this test set will reveal if a model is simply exploiting lexical overlap rather than comparing the underlying concepts. Finally, we run BUMBLEBEE with embedded languages not seen during task-specific training and from a different family (Austronesian) from the XNLI languages, Filipino (tl) and Indonesian (id). This zero-shot defense setting will reveal if CAT encourages the learning of more language-invariant representations, or is simply allowing the model to adapt to the adversarial distribution. Baselines. Since training on languages in the test set takes us out of the cross-lingual transfer setting, we train a translate-trainn baseline for a fair comparison. In this setting, we train on every x and its translations in the n languages sampled in CAT, regardless of whether they contributed words to the final CAT examples. We also include Ganin et al. (2016)'s domain adversarial neural network (DANN), which has been used for cross-lingual adaptation (Joty et al., 2017; Chen et al., 2018). From Table 6, we observe that both training on fully translated data and on CAT examples improved accuracy on the non-adversarial test sets and robustness to code-mixed adversaries, compared to the cross-lingual transfer model that was only trained on English data. Similar to K et al. (2020), we found that disrupting the models' reliance on lexical overlap (Clean DL ) hurt performance. The drop was particularly significant for the cross-lingual transfer (8 points) and translate-trainn models (5.24 points). On the other hand, our CAT model only suffered a 1.5-point drop, indicating that the former two models likely rely heavily on lexical overlap to make predictions, while our CAT model may be using deeper, more language-agnostic features. Crucially, our CAT model achieves similar to better clean accuracy than the baselines, contrasting with prior work showing that adversarial training hurts clean accuracy (Tsipras et al., 2019). Finally, our CAT model is > 1.7x more robust to adversaries constructed from all fifteen XNLI languages than the translate-trainn model. Although DANN-type training improved robustness to the previous BUMBLEBEE adversaries, clean performance was significantly degraded and BUMBLEBEE was able to find even more damaging adversaries upon attacking the model directly. When attacked with 1-2 embedded languages that were seen during training, CAT also yields significant improvements in robustness over the baselines: a > 7 point increase compared to translate-trainn and a > 19 point gain over the zero-shot transfer setting. In the zero-shot defense setting, CAT shows a > 12-point gain over the zero-shot transfer model and a > 4.7-point gain over the translate-trainn model. We believe these results to be due to CAT encouraging the learning of language-invariant representations by exposing the model to cross-lingual lexical variation and preventing the model from exploiting lexical overlaps. To further understand the effect of various fine-tuning methods on XLM-R base , we visualize the <s> vector from the layer before the classification head using t-SNE (Linderman et al., 2019). Here, all sentences from XNLI are passed through the representations individually. If a representation were 100% language-invariant, we should expect t-SNE to be unable to separate individual languages into their own clusters. Hence, the extent to which t-SNE is able to do so would indicate the amount of language-specific information in this last layer. From Fig. 2a, we observe that for the crosslingual transfer model (4), t-SNE managed to organize the sentences from several languages (Chi-nese, Hindi, Thai, Urdu) into distinct clusters. This indicates that a significant amount of language-specific information remains in the vector representations of sentences from these languages. Visualizing the sequence-averaged embeddings makes this even clearer (Fig. 5 in Appendix G). Hence, while XLM-R may be multilingual, it appears to be structured as a space of individual language subspaces as opposed to a mixed, or language-invariant space. On the other hand, t-SNE was much less successful when given the representation trained with CAT (Fig. 2b). Mixing multiple languages in the same sentence and showing the model multiple variants of the same sentence likely encourages the model to refine its representation such that all variants of the same sentence are represented similarly, resulting in a more language-invariant representation. We acknowledge that our methods do not fully model real code-mixing since we do not learn the mixing patterns from real data and there are subtleties in real code-mixing we ignore for simplicity, e.g., accounting for the prestige of participating languages (Bhatia, 2011). In addition, it is impossible to guarantee the semantic preservation of a sentence generated by BUMBLEBEE due to the word aligner's statistical nature, though we can expect more accurate alignments to improve semantic preservation. Finally, while CAT improves robustness, there remains a significant gap between the robust and clean accuracies. In line with recent work challenging the Anglocentricity of crosslingual models (Anastasopoulos and Neubig, 2020; Liang et al., 2020), a promising direction of future work lies in investigating how the choice of matrix language affects model robustness. Ensuring that multilingual models are robust to both natural and adversarial code-mixing is important in today's increasingly multilingual world if they are to allow their target users to fully express themselves in human-machine conversations and to defend against adversarial users attempting to evade toxicity/misinformation detection systems. To approximate a lower bound for model performance on lexical code-mixing, we propose two strong black-box multilingual adversarial attacks and demonstrate their effectiveness on state-of-the-art cross-lingual NLI and QA models. The former generates perturbations from bilingual dictionaries and disambiguates between senses using sentence translations, while the latter generates perturbations by aligning sentences from different languages. Next, we show that training on code-mixed data synthesized via word alignment improves clean and robust accuracy when models are prevented from exploiting lexical overlap without hurting clean accuracy. Crucially, we achieve this in the same number of steps as standard supervised training. Finally, we use t-SNE visualizations to show that multilingual models are not necessarily language-invariant and that our code-mixed adversarial training scheme encourages language-invariance. Adversarial attacks and defenses are double-edged swords. On one hand, adversarial examples expose the gaps in existing models and help to focus the research community's attention on flaws that need to be addressed before these models can be used reliably in noisy, real-world environments. On the other, the same adversarial attacks can be used by malicious actors to bypass toxicity/misinformation detection systems. Similarly, methods for improving adversarial robustness can be used to defend against malicious actors and improve robustness to natural noise or linguistic variation, yet they can also be used to strengthen automated censorship systems and limit freedom of speech. For example, our adversarial attacks could be used both as a lower bound for model performance on naturally occurring code-mixed text and to bypass misinformation detection systems while preserving the message's intelligibility for multilingual speakers. Our adversarial training method could be used to both improve machine understanding of code-mixers by making multilingual representations more language-invariant and suppress the freedom of speech of polyglots who could have been using code-mixing to evade censorship. At the same time, technology strongly shapes our behavior (Reeves et al., 2019). Consequently, given the centrality of code-switching/mixing to many polyglots' lived experiences (Duff, 2015) and the positive correlations between multilingualism, code-switching, and creativity (Leikin, 2013; Kharkhurin and Wei, 2015; Furst and Grin, 2018), we should ensure that the natural language technologies we build do not inhibit multilingual speakers from fully expressing themselves, e.g., by discouraging code-mixing due to non-understanding. In addition, studies have found that aphasic polyglots code-mix more frequently than neurotypical polyglots to cope with word-retrieval difficulties (Goral et al., 2019), making it important for natural language technologies to be robust to code-mixing if they are to be inclusive. Therefore, we include both adversary generation and defense methods to avoid tipping the balance too far in either direction. We would like to thank Cynthia Siew (NUS Psy-chology), Greg Bennett and Kathy Baxter (Sales-force), Lav Varshney (UIUC Electrical and Computer Engineering), Min-Yen Kan (NUS Computer Computer Science), and our anonymous reviewers for their invaluable feedback. We are also grateful to Guangsen Wang, Mathieu Ravaut, Soujanya Lanka, and Tuyen Hoang for contributing manually code-mixed sentences and Bari M Saiful for pointers on replicating the XLM-R results. Samson is supported by Salesforce and Singapore's Economic Development Board under its Industrial Postgraduate Programme.
abstain abstain abstain method abstain abstain abstain objective abstain abstain abstain abstain abstain objective objective objective abstain abstain abstain abstain abstain objective abstain abstain method objective objective objective other other other other other other other other other other other other other other other other other other other other other other other other other other other other method other other objective objective abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain result abstain abstain abstain method abstain abstain method method abstain method method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain method result result abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result result result abstain abstain abstain abstain method abstain abstain method abstain abstain other other other other