Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York . This paper introduces PAWS ( P araphrase A dversaries from W ord S crambling), a new dataset with 108,463 well-formed paraphrase and non-paraphrase pairs with high lexical overlap. Challenging pairs are generated by controlled word swapping and back translation, followed by fluency and paraphrase judgments by human raters. State-of-the-art models trained on existing datasets have dismal performance on PAWS ( < 40% accuracy); however, including PAWS training data for these models improves their accuracy to 85% while maintaining performance on existing tasks. In contrast, models that do not capture non-local contextual information fail even with PAWS training examples. As such, PAWS provides an effective instrument for driving further progress on models that better exploit structure, context, and pairwise comparisons. Word order and syntactic structure have a large impact on sentence meaning. Even small perturbation in word order can completely change interpretation. Consider the following related sentences. Flights from New York to Florida. (2) Flights to Florida from NYC. (3) Flights from Florida to New York. All three have high bag-of-words (BOW) overlap. However, (2) is a paraphrase of (1), while (3) has a very different meaning from (1). Existing datasets lack non-paraphrase pairs like (1) and (3). The Quora Question Pairs (QQP) corpus contains 400k real world pairs, but its negative examples are drawn primarily from related questions. Few have high word overlap, and of the  1,000 pairs with the same BOW, only 20% are not paraphrases. This provides insufficient representative examples to evaluate models' performance on this problem, and there are too few examples for models to learn the importance of word order. Table 1 shows that models trained on QQP are inclined to mark any sentence pairs with high word overlap as paraphrases despite clear clashes in meaning. Models trained or evaluated with only this data may not perform well on real world tasks where such sensitivity is important. To address this, we introduce a workflow (out-lined in Figure 1) for generating pairs of sentences that have high word overlap, but which are balanced with respect to whether they are paraphrases or not. Using this process, we create PAWS ( P araphrase A dversaries from W ord S crambling), a dataset constructed from sentences in Quora and Sentence 1 Sentence 2 Gold BOW BERT BERT+PAWS (1) Can a bad person become good? Wikipedia. Examples are generated from controlled language models and back translation, and given five human ratings each in both phases. A final rule recombines annotated examples and balances the labels. Our final PAWS dataset will be released publicly with 108,463 pairs at https: //g.co/dataset/paws . We show that existing state-of-the-art models fail miserably on PAWS when trained on existing resources, but some perform well when given PAWS training examples. BERT (Devlin et al., 2018) fine-tuned on QQP achieves over 90% accuracy on QQP, but only 33% accuracy on PAWS data in the same domain. However, the accuracy on PAWS boosts to 85% by including 12k PAWS training pairs (without reducing QQP per-formance). Table 1 also shows that the new model is able to correctly classify challenging pairs. Annotation scale is also important: our learning curves show strong models like BERT improve with tens of thousands of training examples. Our experimental results also demonstrate that PAWS effectively measures sensitivity of models to word order and structure. Unlike BERT, a simple BOW model fails to learn from PAWS training examples, demonstrating its weakness at capturing non-local contextual information. Our experiments show that the gains from PAWS examples correlate with the complexity of models. Existing data creation techniques have focused on collecting paraphrases, e.g. from co-captions for images (Lin et al., 2014), tweets with shared URLs (Lan et al., 2017), subtitles (Creutz, 2018), and back translation (Iyyer et al., 2018). Unlike all previous work, we emphasize the collection of challenging negative examples. Our work closely relates to the idea of crafting adversarial examples to break NLP systems. Existing approaches mostly focused on adding label-preserving perturbations to inputs, but with the effect of distracting systems from correct answers. Example perturbation rules include adding noise to inputs (Jia and Liang, 2017; Chen et al., 2018), word replacements (Alzantot et al., 2018; Ribeiro et al., 2018), and syntactic transformation (Iyyer et al., 2018). A notable exception is Glockner et al. (2018): they generated both entailment and contradiction examples by replacing words with their synonyms or antonyms. Our work presents two main departures. We propose a novel method that generates challenging examples with balanced class labels and more word reordering variations than previous work. In addition, we release to public a large set of 108k example pairs with high-quality human labels. We believe the new dataset will benefit future research on both adversarial example generation and improvement of model robustness. In our work, we demonstrate the importance of capturing non-local contextual information in the problem of paraphrase identification. This relates to prior work on probing sentence representations for their linguistic properties, such as how much syntactic information is encoded in representations (Conneau et al., 2018; Tenney et al., { Flights } { from, to } { New York, Florida } LSTM LSTM LSTM LSTM LSTMNNS IN LOCATION (a) Tag words and phrases with part-of-speech (POS) and named entities. (b) Build candidate sets by grouping words and phrases with the same tag. (c) Under the constraints of tag sequence template and candidate sets, find sentences with high language model scores using beam search. 2019; Ettinger et al., 2018). There also exists prior work that directly uses structural information in modeling (Filice et al., 2015; Liu et al., 2018). All these prior approaches were evaluated on existing datasets. In contrast, we perform studies on PAWS, a new dataset that emphasizes the importance of capturing structural information in representation learning. While developing new models is beyond the scope of this paper, this new dataset can facilitate research in this direction. We define a PAWS pair to be a pair of sentences with high bag-of-words (BOW) overlap but different word order. In the Quora Question Pairs corpus, 80% of such pairs are paraphrases. Here, we describe a method to automatically generate nontrivial and well-formed PAWS pairs from real-world text in any domain (this section), and then have them annotated by human raters (Section 4). Our automatic generation method is based on two ideas. The first swaps words to generate a sentence pair with the same BOW, controlled by a language model. The second uses back translation to generate paraphrases with high BOW overlap but different word order. These two strategies generate high-quality, diverse PAWS pairs, balanced evenly between paraphrases and non-paraphrases. Our first phase generates well-formed sentences by swapping words in real world text. Most text generation models rely on large amount of training data (Iyyer et al., 2018; Guu et al., 2018; Gupta et al., 2018; Li et al., 2018), which is unfortunately not available in our case. We thus propose a novel generation method based on language modeling and constrained beam search. The goal is to find a sentence that achieves high language model score as well as satisfying all constraints. High scores indicate that generated sentences are natural and well-formed, and constraints ensure generated pairs have the same BOW. Figure 2 illustrates the generation procedure. First, given an input sentence, a CRF-based part-of-speech tagger tags each word. We further detect person names, locations, and organizations using a named entity recognizer, and replace POS with entity tags if probability scores are above 95%. 1 The sequence of tags of words and phrases form a template for the input. Our beam search method then fills in each slot of the template from left to right, scoring each state by a language model trained on one billion words (Chelba et al., 2014). The candidate words and phrases for each slot are drawn from the input based on its tag. In Figure 2, for example, the second slot must be filled with a LOCATION from two candidate New York and Florida . Candidates are drawn without replacement so the generated sentence and the input have exactly the same bag-of-words. Note that this template-based constraint is more restrictive than the BOW requirement, but we choose it because it significantly reduces the search space. With this constraint, the method achieves high generation quality without a large beam. In practice, beam size is set to 100, which produces near-optimal results in most cases. Let s (cid:48) be the best sentence in the beam other than the input sentence s , and LM (  ) be their log-likelihood by the language model. We take ( s, s (cid:48) ) as a good word-swapping pair if LM ( s (cid:48) )  LM ( s )  t . 2 We manually pick the threshold t =3 . 0 for a good balance between generation quality and coverage. Examples (1) and (2) in Table 2 are representative examples from this generation method. 1 We pick this threshold to achieve about 95% precision. 2 In a preliminary stage, we noticed that many pairs were simply a permutation of a list, like A and B changed to B and A. For the diversity of the dataset, 99% of these are pruned via hand-crafted, heuristic rules. Because word order impacts meaning, especially in English, the swapping method tends to produce non-paraphrases. Our preliminary results showed that the distribution of paraphrase to non-paraphrases from this method is highly imbal-anced (about 1:4 ratio). However, we seek to create a balanced dataset, so we use an additional strategy based on back translationwhich has the opposite label distribution and also produces greater diversity of paraphrases while still maintaining a high BOW overlap. The back translation method takes a sentence pair and label ( s 1 , s 2 , l ) as input. For each sentence, the topk translations are obtained from an English-German neural machine translation model (NMT); then each of these is translated back to English using another German-English NMT model, providing a resulting topk results. We chose German as the pivot language because it produced more word reordering variations than other languages and the translation quality was good. Both models have the same architecture (Wu et al., 2016) and are trained on WMT14. This results in k 2 back translations before deduplication. We chose k =5 . To obtain more pairs with the PAWS property, we further filter back translations by their BOW similarities to the input and their word-order inversion rates, as described below. We define BOW similarity as the cosine similarity  between the word count vectors of a sentence pair. Pairs generated from the swapping strategy have score  = 1 . 0 , but here we relax the threshold to 0.9 because it brings more data diversity and higher coverage, while still generating paraphrases of the input with high quality. To define the word-order inversion rate, we first compute word alignments between a sentence pair in a heuristic way by assuming they are one-to-one mapping and are always monotonic. For example, if the first sentence has three instances of dog and On April 2 Jenkins married Ivy Vujic Jenkins married Ivy on April 2 Figure 3: An example of how to compute inversion rate. the second has two, we align the first two instances of dog in the same order and skip the third one. The inversion rate is then computed as the ratio of cross alignments. Figure 3 is an example pair with six alignments. There are 15 alignment pairs in total and 9 of them are crossed, e.g. alignments of on and married . The inversion rate of this example is therefore 9 / 15 = 0 . 6 . We sample back translation results such that at least half of the pairs have inversion rate over 0.02; this way, the final selected pairs cover interesting transformations of both word-order changes and word replacement. Examples (3) and (4) in Table 2 are representative examples from back translation. Label Balancing Figure 1 illustrates the process of constructing the final label-balanced set based on human annotations. The set first includes all pairs from back translation, which are mostly paraphrases. For each labeled pair ( s 1 , s 2 ) from swapping and a labeled pair ( s 1 , s (cid:48) 1 ) from back translation, the set further includes the pair ( s 2 , s (cid:48) 1 ) based on the rules: (1) ( s 2 , s (cid:48) 1 ) is paraphrase if both ( s 1 , s 2 ) and ( s 1 , s (cid:48) 1 ) are paraphrases; (2) ( s 2 , s (cid:48) 1 ) is non-paraphrase if exactly one of ( s 1 , s 2 ) and ( s 1 , s (cid:48) 1 ) is non-paraphrase; (3) otherwise ( s 2 , s (cid:48) 1 ) is not included because its label is unknown. We also consider pairs ( s (cid:48) 2 , s 1 ) and ( s (cid:48) 2 , s (cid:48) 1 ) in the similar way if s (cid:48) 2 is a back translation of s 2 with human labels. Using the example generation strategies described in Section 3 combined with human paraphrase an-Quora notations, we create a large new dataset, PAWS that contains both paraphrase and non-paraphrase pairs that have both high bag-of-words overlap and word reordering. Source sentences are drawn from both the Quora Question Pairs (QQP) corpus (Iyer et al., 2017) and Wikipedia. 3 From these, we produce two datasets, PAWSQQP and PAWS Wiki . We start by producing swapped examples from both QQP and Wikipedia. Both sources contain naturally occurring sentences covering many top-ics. On both corpora only about 3% of candidates are selected for further processingthe rest are filtered because there is no valid generation candidate that satisfies all swapping constraints or because the language model score of the best candidate is below the threshold. The remaining pairs (16,280 for QQP and 50k for Wikipedia) are passed to human review. Sentence correction The examples generated using both of our strategies are generally of high quality, but they still need to be checked with respect to grammar and coherence. Annotators evaluate each generated sentence without seeing its source sentence. The sentence is accepted as is, fixed, or rejected. Table 3 shows the number of pairs of each action on each domain. Most of fixes are minor grammar corrections like a apple  an apple . Accepted and fixed sentences are then passed to the next stage for paraphrase annotation. Total # back translation pairs 26,897 paraphrase 25,521 non-paraphrase 1,376 Human agreement 94.8% Paraphrase identification Sentence pairs are presented to five annotators, each of which gives a binary judgment as to whether they are paraphrases or not. We choose binary judgments to make our dataset have the same label schema as the QQP corpus. Table 3 shows aggregated annotation statistics on both domains, including the number of paraphrase (positive) and non-paraphrase (negative) pairs and human agreement, which is the percentage ratio of agreement between each individual label and the majority vote of five labels on each example pair. Overall, human agreement is high on both Quora (92.0%) and Wikipedia (94.7%) and each label only takes about 24 seconds. As such, answers are usually straightforward to human raters. To ensure the data is comprised of clearly paraphrase or non-paraphrase pairs, only examples with four or five raters agreeing are kept. 4 An example of low agreement is Why is the 20th-century music so different from the 21st music? v.s. Why is the 21st century music so different from the 20th century music? , where three out of five raters gave negative labels on this pair. The bottom block of Table 3 shows the final number of pairs after this filtering, and human agreement further goes up to over 95%. Finally, source and generated sentences are randomly flipped to mask their provenance. The swapping strategy generally produces non-paraphrase examples67% for QQP and 88% for Wikipedia. Because (a) the label imbalance is less pronounced for QQP and (b) NMT models perform poorly on Quora questions due to domain mismatch, we only apply the back translation strategy to Wikipedia pairs. Doing so creates 26,897 candidate example pairs after filtering. As before, each pair is rated by five annotators on the paraphrase identification task. 5 Table 4 shows that 4 We exclude low agreement pairs from our experiments, but we include them in our data release for further study. most of the examples (94.9%) are paraphrases (as expected), with high human agreement (94.8%). Finally, we expand the pairs using the the rules described in Section 3.2. Table 5 provides counts for each split in the final PAWS datasets. The training portion of PAWSQQP is a subset of the QQP training set; however, PAWSQQP 's development set is a subset of both QQP's development and test sets because there are only 677 pairs. PAWS Wiki randomly draws 8,000 pairs for each of its development and test sets and takes the rest as its training set, with no overlap of source sentences across sets. Finally, any trivial pairs with identical sentences from development and test sets are removed. 6 The final PAWSQQP has a total of 12,665 pairs (443k tokens), where 31.3% of them have positive labels (paraphrases). PAWS Wiki has a total of 65,401 pairs (2.8m to-kens), where 44.2% of them are paraphrases. Note that we have human annotations on 43k pairs generated by the word swapping method on Wikipedia, but 30k of them have no back translation counterparts and therefore they are not included in our final PAWS Wiki dataset. Nevertheless, they are high-quality pairs with manual labels, so we include them as an auxiliary training set (PAWS Wiki-Swap in Table 5), and empirically show its impact in Section 6. Unlabeled PAWS Wiki In addition to the fully labeled PAWS Wiki dataset, we also construct an unlabeled PAWS Wiki set at large scale. The idea is to simply treat all pairs from word swapping as non-paraphrases and all pairs from back translation as paraphrase, and construct the dataset in the same way as labeled PAWS Wiki . The result is a total of 656k pairs with silver labels. We show empirically NMT generates fluent output. 6 Such trivial examples exist because annotators sometimes fix a swapped sentence back to its source. We keep such examples in the training set (about 8% of the corpus) because otherwise a trained model would actually predict low similarity scores to identical pairs. PAWS is designed to probe models' ability to go beyond recognizing overall sentence similarity or relatedness. As noted in the introduction, modelseven the best avaliabletrained on existing resources tend to classify any example with high BOW overlap as a paraphrase. Can any of these models learn finer structural sensitivity when provided with PAWS examples as part of their training? We consider six different models that cover a wide range of complexity and expressiveness: two baseline encoders and four recent advanced models that achieved state-of-the-art or strong performance on paraphrase identification. Table 6 summarizes the models with respect to whether they represent non-local contexts or support cross-sentential word interaction. The baseline models use cosine similarity with simple sentence encoders: a bag-of-words ( BOW ) encoder based on token unigram and bigram encodings and a bi-directional LSTM ( BiLSTM ) that produces a contextualized sentence encoding. A cosine value above .5 is taken as a paraphrase. ESIM . The Enhanced Sequential Inference Model (Chen et al., 2017) achieved competitive performance on eight sentence pair modeling tasks (Lan and Xu, 2018). It encodes each sentence using a BiLSTM, concatenates the encodings for each sentence in the pair, and passes them through a multi-layer perceptron (MLP) for classification. The additional layers allow ESIM to capture more complex sentence interaction than cosine similarity in the baseline models. DecAtt . The Decomposable Attention Model (Parikh et al., 2016) is one of the earliest models to introduce attention for paraphrase identification. It computes word pair interaction between two sentences and aggregates aligned vectors for final classification. This model achieved state-of-the-art results without explicitly modeling word order. In our experiments, we show the limitations of this modeling choice on PAWS pairs. DIIN . The Densely Interactive Inference Network (Gong et al., 2018) adopts DenseNet (Huang et al., 2017), a 2-dimensional convolution architecture, to extract high-order word-by-word interaction between n-gram pairs. This model achieved state-of-the-art performance without relying on pre-trained deep contextualized representations like ELMo (Peters et al., 2018). It outperformed ESIM and DecAtt models by a large margin on both paraphrase identification and natural language inference tasks. BERT . The Bidirectional Encoder Representations from Transformers (Devlin et al., 2018) recently obtained new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement). BERT involves pretraining a Transformer encoder (Vaswani et al., 2017) on a large corpus with over three billion words. This large network is then fine-tuned with just one additional output layer. We seek to understand how well models trained on standard datasets perform on PAWS pairs and to see which models are most able to learn from PAWS pairs. A strong model should improve significantly on PAWS when trained on PAWS pairs without diminishing performance on existing datasets like QQP. Overall, both DIIN and BERT prove remarkably able to adapt to PAWS pairs and perform well on both PAWSQQP and PAWS Wiki while the other models prove far less capable. We use two metrics: classification accuracy and area-under-curve (AUC) scores of precision-recall curves. For all classification models, 0.5 is the threshold used to compute accuracy. We report results on testing sets for QQP and PAWS Wiki , and on the development set for PAWSQQP (which has no test set). For BERT, we use the implementation provided by the authors 7 and apply their default fine-tuning configuration. We use the provided BERTBASE pre-trained model instead of BERTLARGE due to GPU memory limitations. For all other models, we use our own (re-)implementations that matched 7 https://github.com/google-research/bert reported performance on QQP. We use 300 dimensional GloVe embeddings (Pennington et al., 2014) to represent words and fix them during training. Main Results on PAWSQQP Table 7 summarizes results on the Quora domain. We first train models on the Quora Question Pairs (QQP) training set, and column QQP  QQP shows that all models achieve over 83% accuracy on QQP. However, when evaluating on PAWSQQP , all models, including BERT, obtain abysmal accuracy under 40% (column QQP  PAWSQQP ). We hypothesize the performance on PAWSQQP relies on two factors: the number of representative training examples, and the capability of models to represent complex interactions between words in each sentence and across the sentences in the pair. To verify that, we further train models on a combination of QQP and PAWSQQP training sets and the last two columns of Table 7 show the results on PAWSQQP . As expected, all models benefit from new training examples, but to different extents. Gains are much larger on state-of-the-art models like BERT, while the BOW model learns almost nothing from new examples. As a consequence, performance changes are more drastic on PAWSQQP than on QQP. For example, the absolute difference between BiLSTM and BERT is 4.2% on QQP, but it goes up to 27% on PAWSQQP , which is a 60% relative reduction in error. It is also noteworthy that adding PAWSQQP training examples has no negative impact to QQP performance at all. For example, a BERT model fine-tuned on QQP+PAWS QQP achieves the same 90.5% classification accuracy as training on QQP alone. We therefore obtain a single model that performs well on both datasets. Main Results on PAWS Wiki In our second experiment we train and evaluate models on our PAWS Wiki dataset. Table 8 presents the results. DIIN and BERT outperform others by a substantial margin ( > 17% accuracy gains). This observation gives more evidence that PAWS data effectively measures models' sensitivity to word order and syntactic structure. One interesting observation is that DecAtt performs as poorly as BOW on this dataset. This is likely due to the fact that DecAtt and BOW both consider only local context information. We there-M ODELSQQP  QQP QQP  PAWSQQP QQP+PAWS QQP  PAWSQQP (Acc) (AUC) (Acc) (AUC) (Acc) (AUC) BOW 83.2 89.5 29.0 27.1 30.0 (+1.0) 27.3 (+0.2) BiLSTM 86.3 91.6 34.8 37.9 57.6 (+22.9) 52.3 (+14.5) ESIM (Chen et al., 2017) 85.3 92.8 38.9 26.9 66.5 (+27.7) 48.1 (+17.2) DecAtt (Parikh et al., 2016) 87.8 93.9 33.3 26.3 67.4 (+34.1) 51.1 (+24.9) DIIN (Gong et al., 2018) 89.2 95.2 32.8 32.4 83.8 (+51.1) 77.8 (+45.5) BERT (Devlin et al., 2018) 90.5 96.3 33.5 35.1 85.0 (+51.5) 83.1 (+48.0) Table 7: Acc uracy (%) of classification and AUC scores (%) of precision-recall curves on Quora Question Pairs ( QQP ) testing set and our PAWSQQP development set. fore tested an enhancement of DecAtt by replacing its word representations with encodings from a BiLSTM encoder to capture non-local context information. The enhanced model significantly outperforms the base, yielding an 11.5% (57.1% vs. 68.6%) absolute gain on accuracy. We further evaluate the impact of using silver PAWS Wiki data in pre-training, as discussed in Section 4. The last two columns of Table 8 show the results. Comparing to supervised performance, pre-training with silver data gives consistent improvements across all models except BOW and vanilla DecAtt. Perhaps surprisingly, adding silver data gives more than 10% absolute improvements on AUC scores for BiLSTM and ESIM, much higher than the gains on DIIN and BERT. train multiple models on QQP plus different number of PAWSQQP examples. Figure 4 plots AUC score curves of DIIN and BERT as a function of the number of PAWSQQP training examples. x = 0 corresponds to models trained on QQP only, and the rightmost points correspond to models trained on QQP and full PAWSQQP . Both models improve from 30% to 74% AUC scores with 6,000 PAWSQQP examples. Furthermore, neither curve reaches convergence, so they would likely still benefit from more PAWS training examples. Cross-domain Results The PAWS datasets cover two domains: Quora and Wikipedia. Here we demonstrate that a model trained on one domain also generalizes to another domain, although not as well as training on in-domain data. Table 9 shows that a DIIN model trained on Quora (QQP+PAWS QQP ) achieves 70.5% AUC on the Wikipedia domain. This is lower than training on in-domain data (92.9%), but higher than the model trained without any PAWS data (46.0%). We also observe similar patterns when TRAININGDATAQQP PAWSQQPPAWS Wiki (Test) (Dev) (Test) QQP (Train) 95.2 32.4 46.0 QQP+PAWS QQP 95.3 77.8 70.5 QQP+PAWS Wiki 95.3 58.5 92.9 +PAWS Wiki-Swap 95.3 70.6 93.5 QQP+PAWS QQP+Wiki 95.1 87.0 93.4 +PAWS Wiki-Swap 95.3 89.9 93.8 Table 9: AUC scores (%) when training DIIN models on different sets of training data. training on Wikipedia (QQP+PAWS Wiki ) and testing on PAWSQQP . Interestingly, using out-of-domain data also boosts in-domain performance. As Table 9 shows, training on both domains (QQP+PAWS QQP+Wiki ) leads to 9.2% absolute AUC gains on PAWSQQP over the model trained only on QQP+PAWS QQP . The auxiliary training set on Wikipedia (PAWS Wiki-Swap ) helps further. As Table 9 shows, adding this auxiliary training set is particularly helpful to the performance on PAWSQQP , yielding a 12.1% (70.6% vs 58.5%) gain on AUC when training on QQP+PAWS Wiki . On PAWS Wiki , this addition lifts the (no pre-training) DIIN model AUC from 91.1% (Table 8) to 93.8% (Table 9). BERT vs DIIN Both models achieve top scores on PAWS, but interestingly, the two models disagree on many pairs and are not correlated in their errors. For example, of 687 of BERT's mistakes on the PAWS Wiki test set, DIIN got 280 (41%) correct. As such, performance might improve with combinations of these two existing models. It is also worth noting that the DIIN model used in our experiments has only 590k model parameters, whereas BERT has over 100m. Furthermore, the computational cost of BERT is notably higher than DIIN. Given this, and the fact that DIIN is competitive with BERT (especially when pre-trained on noisy pairs, see Table 8), DIIN is likely the better choice in computationally constrained scenariosespecially those with strict latency requirements. Datasets are insufficient for differentiating models if they lack examples that exhibit the necessary diagnostic phenomena. This has led, for example, to new datasets for noun-verb ambiguity (Elkahky et al., 2018) and gender bias in coreference (Web-ster et al., 2018; Rudinger et al., 2018; Jieyu Zhao, 2018). Our new PAWS datasets join these efforts and provide a new resource for training and evaluating paraphrase identifiers. We show that including PAWS training data for state-of-the-art models dramatically improves their performance on challenging examples and makes them more robust to real world examples. We also demonstrate that PAWS effectively measures sensitivity of models on word order and syntactic structure. We would like to thank our anonymous reviewers and the Google AI Language team, especially Emily Pitler, for the insightful comments that contributed to this paper. Many thanks also to the Data Compute team, especially Ashwin Kakarla and Henry Jicha, for their help with the annotations References Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain other result abstain abstain abstain result objective abstain result other other objective abstain other other other method objective abstain objective objective other other other other other other other objective objective method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain method method method other method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result objective other other