Controlled text perturbation is useful for evaluating and improving model generalizability. However, current techniques rely on training a model for every target perturbation, which is expensive and hard to generalize. We present T ailor , a semantically-controlled text generation system. T ailor builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from semantic representations. We craft a set of operations to modify the control codes, which in turn steer generation towards targeted attributes. These operations can be further composed into higher-level ones, allowing for flexible perturbation strategies. We demonstrate the e  ectiveness of these perturbations in multiple applications. First, we use T ailor to automatically create high-quality contrast sets for four distinct natural language processing (NLP) tasks. These contrast sets contain fewer spurious artifacts and are complementary to manually annotated ones in their lexical diversity. Second, we show that T ailor perturbations can improve model generalization through data augmentation. Perturbing just  2% of training data leads to a 5.8-point gain on an NLI challenge set measuring reliance on syntactic heuristics. Semantic perturbation through controlled text generation modifies sentences to match certain target attributes, such as verb tense or sentiment ( e.g., positive  negative ). It has been widely applied to a variety of tasks, e.g., changing text style (Reid and Zhong, 2021), mitigating dataset biases (Gardner et al., 2021), explaining model behaviors (Ross et al., 2021), and improving model generalization (Teney et al., 2020; Wu et al., 2021). Existing e  orts train task-specific generators, e.g., training  denotes equal contribution. a sentiment style transferer requires instances annotated with positive and negative labels (Madaan et al., 2020b). As a result, they require costly annotated data and re-training for every task of interest. This work introduces T ailor , a system that supports application-agnostic perturbations. At its core is a controlled generator (2) that flexibly generates outputs from target semantic attributes, which we represent through structured control codes in the inputs. As shown in Figure 1, these control codes build on the PropBank semantic analysis (Palmer et al., 2005) of the original sentence: For each argument span, the semantic role and keyword control codes specify the desired semantic content for the span at varying levels of granu-1 We opensource T ailor and release T ailor -generated contrast sets at https://github.com/allenai/tailor . larity. To encourage control code following, we train the T ailor generator with unlikelihood training (Welleck et al., 2020) to penalize generations that are not aligned with designated control codes. The use of semantic role control codes allows T ailor to perform fine-grained changes to individual arguments in a sentence ( e.g., one can change only the PATIENT in Figure 1). Instead of specifying a perturbation with a generic target property ( e.g., positive (cid:41) negative ), we can specify the linguistic transformation used to achieve the property ( e.g., changing sentiment through negation or antonym replacement). Making such fine-grained perturbations allows for more careful evaluation and improvement of models' language understanding (Kaushik et al., 2020; Wu et al., 2021). To highlight the perturbations facilitated by T ai lor , we craft a list of primitive perturbation operations (3) on inputs to the generator; these can be easily composed to achieve more complex perturbations. In Figure 1, T ailor transforms sentence A to D through a series of perturbations: syntactic rewriting (changing verb tense), then sentence expansion (extending the athlete), and finally data recombination ( i.e., generating new text that contains in but follows the TEMPORAL control). Compared to existing approaches that require training a separate model for every step or annotating a dataset that represents this transformation end-to-end, such compositions make T ailor more cost-e  ective and generalizable. In fact, on nine fine-grained and compositional S tyle PTB perturbations (Lyu et al., 2021), T ailor achieves performance compatible with task-specific baselines, and even outperforms them on five transfers (F). T ailor 's flexible and human-readable control codes allow for broad, easily extendable applicability. We demonstrate its utility in evaluating and improving NLP model robustness, showing that T ailor can help replicate existing contrast sets on four diverse tasks. By abstracting manual perturbation types in prior work into perturbation strategies with T ailor , we can apply the changes to larger datasets while saving manual annotation e  orts. Our analysis suggests that these contrast sets not only have high rates of validity, but also reduce spurious artifacts compared to the original evaluation datasets. In addition, T ailor -produced contrast sets complement human annotated ones in terms of lexical diversity: only  10% of their unique tokens overlap with manually created contrast sets. We also explore T ailor 's utility in data augmentation. We find that augmenting training data with just  2% of T ailor perturbations improves the robustness of natural language inference (NLI) models to inference heuristics, increasing performance on the HANS evaluation set (McCoy et al., 2019) by an average of 5.81 points and outperforming a previous syntactic augmentation method for NLI. Here, we provide an overview of the T ailor generator. We first outline three types of controls (2.1) that allow for specifying sentence meanings at varying granularity. Next, we explain how to embed them within inputs (2.2) to the generator. We train the generator to follow control codes with unlikelihood training (2.3). We use the following three types of controls to specify the shallow semantics, actual content, and ordering of various phrases in a sentence. Semantic roles to denote shallow semantics. We rely on the PropBank semantic formalism (Palmer et al., 2005), as it provides well-established representations of meanings that are generalizable across di  erent predicates and languages (Hajic et al., 2009). It represents sentence meanings with predicate-argument structures. Predicates ( e.g., comforted in Figure 1) are usually evoked by verbs and reflect events ( what happened ). Arguments, usually spans of tokens, realize thematic roles of predicates; they include core arguments such as who did something ( e.g., the doctor in Figure 1) and to whom (the athlete), as well as adjunct arguments like where something happened (In the operation room) and how . Keywords for steering the actual generated content of predicates and arguments. The keywords can be complete and fully specify the target text of a given span ( e.g., the doctor for the AGENT in Table 1A), sparse and add no constraints beyond the semantic role ( e.g., * for LOCATIVE ), or partial and specify some of the target text ( e.g., athlete for PATIENT ). As later shown in Table 3, these keyword controls are important for supporting a variety of perturbation strategies and applications. Span ordering for determining how the thematic roles should be combined. We use predicate form to control the order of core arguments. For example, to distinguish the athlete was comforted by 3195 Input Target Output Description A [ VERB + active + past : comfort | AGENT + complete : the doctor | PATIENT + partial : athlete | LOCATIVE + sparse : *] <id_0> , <id_1> <id_2> <id_3> . [ LOCATIVE : In the operating room], [ AGENT : the doctor] [ VERB : comforted] [ PATIENT : the athlete]. Mask all roles B [ VERB + active + past : comfort | LOCATIVE + sparse : *] <id_0> , the doctor <id_1> <id_2> the athlete <id_3> . [ LOCATIVE : In the operating room], the doctor [ VERB : comforted] the athlete. Empty blanks C [ VERB + active + past : comfort | LOCATIVE + sparse : *] <id_0> , the doctor <id_1> the athlete. [ LOCATIVE : In the operating room], the doctor [ VERB : comforted] the athlete. Mask subset of arguments N [ VERB + passive + present : comfort | PATIENT + complete : the doctor | AGENT + partial : athlete | TEMPORAL + sparse : *] <id_0> , <id_1> <id_2> <id_3> . [ TEMPORAL : In the operating room], [ PATIENT : the doctor] [ VERB : comforted] [ AGENT : the athlete]. Negative sample Table 1: Example input / output formats for sentence In the operating room, the doctor comforted the athlete. the doctor from the semantically equivalent the doctor comforted the athlete, we target the former ordering through a passive control, and the latter through an active control. Additionally, we use the location of blank tokens ( <id_*> in Figure 1 and Table 1) to determine the position of generated arguments (Wu et al., 2021)  e.g., where in the operating room appears in the generation. We integrate the aforementioned controls into the input format detailed in A.1 and finetune seq2seq models to output corresponding full sentences. As shown in Table 1, we start our input with a bracketed header , which contains a series of abstract control codes (Table 2) that denote the semantic role and keywords (content / specificity) to realize for each predicate and argument. For example, in Table 1A, the control codes for the predicate are  VERB + active : past and the agent argument are  AGENT + complete : the doctor. We 2 We use http://spacy.io/ for verb or POS detection. map original semantic roles in PropBank to human-readable labels ( i.e., ARG0  AGENT ) in order to leverage knowledge learned by pretrained models about roles' meanings (Paolini et al., 2021). After the header, we append the context , which consists of text to preserved and blanks specifying where new text should be generated. Given such inputs, we train our generator to output text augmented with control codes and brackets, which together specify which generated spans correspond to which controls. For example, in Table 1B, [ LOCATIVE : In the operating room] represents the target span of control codes  LOCATIVE + sparse : * and is generated at the location of blank <id_0> right before the preserved context the doctor. We make three key design choices to allow T ai lor to generate roles fluently even when the optimal ordering of roles is unknown ( e.g., when introducing a new argument). First, we explicitly separate signal about role placement ( e.g., blanks in the context) from the role's semantic controls ( e.g., control codes in the header) such that we can specify the target semantic attributes for a role without tying them to a specific target placement. Second, we order the control codes in the header in an input-independent way (see A.1) to discourage the generator from learning to rely on their relative orders. Third, we insert extra empty blanks into the context ( e.g., <id_3> in Table 1B) such that the T ailor generator can generate spans in the blank locations that result in the most fluent text. With this flexibility in argument ordering comes the challenge of making strict controls on a single argument: Even if we only want to change verb tense, the generator may reorder other arguments. To enable strict control over generations, which facilitates minimal perturbations (Ross et al., 2021), we further vary the number of arguments encoded in the header. As in Table 1C, our generator can take inputs that only mask a subset of arguments, such that, e.g., any changes on the LOCATIVE argument or VERB do not a  ect the agent and patient. We finetune T5base (Ra  el et al., 2020) on input-output pairs derived from gold semantic roles in OntoNotes 5.0 train (Table 1; Pradhan et al., 2013). 3 To train our generator to handle the different input formats described in 2.2, for each original input, we randomly sample the numbers of arguments to mask, number and placement of extra empty blanks, and keyword content / specificity for each role. See A.2 for details. Standard maximum likelihood estimation (MLE) is insu  cient for training our generator to follow the controls, as there may exist signals beyond the given controls for the form of a generation. Consider the input: [ VERB + active + past : comfort | AGENT + partial : athlete | PATIENT + complete : the doctor] In the operating room, <id_0> , <id_1> <id_2> . A generator trained with MLE may ignore controls AGENT and PATIENT and instead output text The doctor comforted the athlete rather than The athlete comforted the doctor, as the training data distribution may reflect that the former is more natural given context in the operation room. To encourage reliance on controls, we incorporate unlikelihood training (Welleck et al., 2020) to penalize generations that conflict with input controls. That is, besides Table 1AC which are used for MLE, we also create negative samples by randomly perturbing the control codes in our header (as in Table 1N, last row), such that most spans in the target output are not aligned with the control codes. We create up to three negative samples per input by randomly perturbing 1) verb voice / tense and primary controls for arguments, 2) keyword contents, and 3) keyword specificities (A.1).Our final training data consists of 223K positive and 541K negative examples. With T ailor , we can create diverse perturbations by modifying input controls. Given an original 3 On par with T5, the blanks are in the form of <extra_id_*> ; we refer them as <id_*> for simplicity. sentence, we transform it to an input for T ailor by extracting its semantic parses, 4 masking spans we wish to modify, and providing their control codes. Then, we modify the control codes in the input to generate perturbed sentences with T ailor , filtering out degenerate ones. Primitive perturbation operations. We provide an easily-extendable set of perturbation macros, which capture three common types of perturbations in prior work, shown in Table 3: First, syntactic rewriting primarily involves shu  ing text to create paraphrases (Zhang et al., 2019) or adversarial examples (Iyyer et al., 2018). We implement such shu  ing through operations that perturb predicate forms, move blank tokens, and swap keyword contents of arguments. Second, expansion and abstraction add or remove text fragments from a sentence (Wu et al., 2021). We recreate these through operations on keywords ( e.g., dele-tion). Finally, data recombination involves recombining existing textual fragments, within or across inputs (Akyrek et al., 2021; Andreas, 2020). With CHANGE_CONTENT , we can integrate additional context ( e.g., from corresponding paragraphs in question answering tasks) into generations. While our control codes are mostly derived from semantic roles, these primitive operations broadly cover both syntactic and semantic changes. They can also be used in conjunction with external knowledge bases to achieve targeted edits. 5 , or be composed to achieve more complex perturbation strategies as shown in 5, 6, and Appendix F. Filtering generations. We notice that the T ailor generator produces degenerate outputs for some inputs; we exclude these heuristically based on content and perplexity scores (see C for details). Following previous work (Wu et al., 2021; Ross et al., 2021), we evaluate T ailor generations on sentence likelihood, controllability, and closeness. 6 4 External semantic role labelers can be used when gold annotations are not available. Our experiments use the opensourced implementation of Shi and Lin (2019): demo. allennlp.org/semantic-role-labeling , with a test F1 of 86.5 on the Ontonotes 5.0 dataset (Pradhan et al., 2013). 5 For example, if combined with WordNet (Miller, 1998), T ailor perturbations may be able to incorporate a subset of natural logic (MacCartney and Manning, 2014): In Figure 1, we can create an entailment relationship by replacing doctor with its hyponym adult. We additionally evaluate T ailor 's unique ability to make fine-grained and compositional perturbations. Metrics. Likelihood measures whether the generated text is grammatically correct and semantically meaningful. Following Ross et al. (2021), we ask whether perturbing a sentence with T ailor drastically changes its likelihood. Using a pretrained GPT-2, we compute language modeling losses for both the original and edited texts and report the ratio of edited / original. We desire a value of 1.0, which indicates equivalent losses for the two. Controllability measures if the generator responds to the controls given in inputs. We rely on cycle consistency to evaluate the controls in Table 2: For a given generation, we check whether the predicted semantic roles from an SRL system match the control codes in the input ( e.g., whether in the midst of the earthquake in Figure 1 gets detected with a TEMPORAL tag). Since SRL predictions can be noisy, we manually inspect a subset of 98 generated spans and verify that cycle consistency measures positively correlate with groundtruth controllability, with Matthews correlation coe  cient  = 0 . 49 (more details in B). Closeness captures whether the generated sentence involves only necessary changes. Since our generator takes controls at the argument level, we measure closeness with a weighted F1 score on the expected-to-change and actually-changed spans in the original sentence. We identify expected-to-change spans from perturbation operations; in Figure 1A, all spans should be changed except for agent the doctor. Then, we deem a span actually edited if  50% tokens within a span are changed ( e.g., operation room in LOCATIVE ). 7 We weigh spans by their lengths to arrive at the final F1 score. Compositionality. We evaluate T ailor without any finetuning on the S tyle PTB benchmark (Lyu et al., 2021), which builds on the Penn Treebank and assesses both single , fine-grained transfers ( e.g., To Future Tense ) and compositional ones that concurrently edit multiple dimensions ( e.g., To Future Tense + Active To Passive ). We report mean BLEU scores and compare to the transfer-specific baselines reported in the S tyle PTB paper (See F). Data. We use S tyle PTB (Lyu et al., 2021) to evaluate compositionality. For other metrics, we perturb 1,000 randomly selected sentences from the OntoNotes 5.0 validation dataset, created the same way as negative samples during training (A.1), and evaluate on these perturbations. 8 7 We empirically tune the threshold to be 50%, as it tolerates cases where we do not know exactly how the tokens should change ( e.g., when changing keyword sparsity, we do not know exactly how many new tokens should be generated; when changing semantic role controls, we may want to allow some tokens, like particles, to reoccur, while expecting others in the span to change.) 8 Because these perturbations are generated randomly, some result in sets of controls that are impossible to follow. Thus, these results represent a lower bound on T ailor 's controllability in downstream applications, for which strategies would be designed in a more principled, targeted manner, re-3198 4.1 Results T ailor generates perturbations with a loss ratio of 0.982, indicating no notable change in language modeling loss after perturbation. As shown in Table 4, T ailor perturbations also tend to be close to the original sentence (F1 = 64 . 3%), with reasonably correct predicates (74.3%-81.6% of the time) and arguments (70.5% controllability on semantic roles and 64.5% on contents.) T ailor also demonstrates the ability to make compositional changes; it achieves results comparable to those of fine-tuned baselines on 8 / 9 tested transfers, and even outperforms the fine-tuned baseline on 5 of them (See F and Table 11 for more details). E  ect of Unlikelihood Training. We compare T ailor with a baseline that is finetuned on T5 without unlikelihood training (called T ailor MLE in Table 4). Across all metrics, unlikelihood training outperforms T ailor MLE , with more controllable and closer perturbations (up to a 20% increase). Modulating likelihood and closeness. As mentioned in 2.2, our input format supports modulating likelihood and closeness. We can increase closeness by only masking the arguments we want to perturb. To quantify this e  ect, we randomly select a single argument to perturb for 1K sentences, but vary the number of masked arguments and number of inserted blanks. As desired, closeness is maximized when we mask only the argument we wish to perturb, as in Table 1B (with F 1 = 67 . 4%), whereas masking two extra arguments and inserting six extra blanks decreases closeness by 3% and 6%, respectively. On the other hand, we can prioritize likelihood (at the cost of closeness) by adding more blanks ( e.g., insert extra roles whose optimal locations are not known in advance). On another 1K sentences, we observe that adding six extra blanks increases the likelihood ratio from 0.93 to 0.95. Manually creating contrast sets is expensive, e.g., Gardner et al. (2020) reported spending 10-15 minutes per perturbation for UD Parsing, whereas labeling existing data is more e  cient (Wu et al., 2021). We show that T ailor can reduce human labor by automatically generating contrast set instances such that annotators only have to label them. We create T ailor -generated contrast sets for four stricting the perturbations to result in more plausible sets of controls. See B for more details. tasks: boolean question answering (BoolQ: Clark et al., 2019), extractive QA (SQuAD: Rajpurkar et al., 2016), dependency tree parsing (UD English: Nivre et al., 2016), and temporal relation extraction (MATRES: Ning et al., 2018). 9 5.1 Replicating Contrast Sets with T ailor We take advantage of two key properties of T ai lor : First, T ailor can make context-dependent changes. To recreate the BoolQ contrast set , we replicate Entity Change in Gardner et al. (2020) by replacing content keywords in questions with words in the paragraph that have the same semantic roles. For example, the paragraph in Table 5 indicates that his bride can serve as an AGENT . Second, T ailor allows for compositonal changes. For example, as in Table 5, we change prepositional phrase (PP) attachments from noun  verb to recreate the UD Parsing contrast set through the following composition of perturbation operations: remove the prepositional phrase from the patient keyword ( e.g., a diverse range of food at all prices and styles), and introduce an adjunct argument with the preposition as partial keyword ( e.g., LOCATIVE at). More details are in D.1. Contrast set validity. We consider our perturbation strategies successful if they help reduce human labor, i.e., a contrast set author can easily label or take inspiration from T ailor 's generations. Two authors sampled 100 original instances per task, inspected the top-K T ailor perturbations, and labeled an instance to be valid if there is at least one perturbation that changes the groundtruth answer while being fluent or requiring only minor fixes. 10 Table 5 shows that these T ailor perturbation strategies generate contrast sets with high validity. 11 5.2 Measuring Contrast Set Quality We sanity check that T ailor -generated contrast sets can be used to reveal model errors. For example, a T5base model finetuned on BoolQ (with test accuracy 83%) has a performance of 65% on both T ailor -generated contrast sets and Gardner et al. (2020)'s (more in D.2). However, this metric is 9 T ailor -generated contrast sets are available at https: //github.com/allenai/tailor . 10 Because we exercised controls at di  erent granularity ( i.e., UD requires sourcing contents from the generator while others mostly require syntactic rewrites with predetermined content), we set k = 10 for UDan upper bound for not overloading the human inspectorand k = 1 for other tasks. 11 T ailor achieves higher validity changing attachment from noun  verb (82%) than verb  noun (48%). only a proxy for the quality of evaluation data, since it can be made intentionally low if we generate all examples to target a known model error. Thus, we directly analyze the quality of T ailor contrast sets by measuring their lexical diversity and impact on token-level dataset artifacts , both of which play important roles in dataset debiasing. We measure lexical diversity on UD Parsing contrast sets because it involves su  cient generation of new content. We compare T ailor and human-generated (Gardner et al., 2020) contrastive edits for the same 100 UD instances: we randomly sample one edit for each valid instance, heuristically extract modified PPs, and compute diversity as the ratio of unique to total new tokens in the PPs, filtering stopwords. For noun  verb , the ratios are respectively 0.78 and 0.99 for T ailor and humans; for verb  noun , both are 1.0. Thus, T ailor can help generate contrast sets without significantly reducing lexical diversity. Furthermore, T ailor outputs are distinguishable from humans': their unique tokens only overlap for < 15% in verb  noun , and  6% for noun  verb , suggesting that T ailor can be used as a collaborative tool to diversify generation. We also ask whether T ailor perturbations can reduce dataset artifacts. Gardner et al. (2021) devise a statistical test for dataset artifacts that builds on the argument that no simple feature ( e.g., single token) should show statistically significant correlation with labels in a language understanding problem. In Figure 2, we display the results: We plot the numbers of occurrences of each token against the conditional probability of the positive label given that token for both the BoolQ validation data (red dots) and the contrast created by T ailor (green dots). All tokens above or below the blue line show statistically significant correlation with positive labels and thus are considered dataset artifacts in Gardner et al. (2021)'s framework. While many tokens in the original BoolQ data exhibit significant correlations, most in the T ailor contrast set fall within the confidence region. Thus, T ailor can help create less evaluation data with fewer artifacts. Across the four tasks, we are able to replicate all perturbation strategies described by authors of the original contrast sets. While T ailor requires manual e  ort to implement perturbation strategies, we believe the overall saved annotation effort outweighs this initial cost. First, once implemented, T ailor perturbations can be applied to large datasets without requiring additional annotation e  ort. This large-scale applicability is especially useful for tasks whose single-instance annotation time is significant ( e.g., UD Parsing). Second, given that T ailor generations are distinguishable from human ones, they may have the potential to compensate for human omissions and thereby increase test case variety, which has been shown to be beneficial in prior work (Ribeiro et al., 2020); an interesting direction for future work would be to investigate this hypothesis in more detail. Third, the implementation overhead itself diminishes as more strategies are implemented. In BoolQ, while Gardner et al. (2020) manually created a diverse set of perturbations, including adjective, entity, and event changes (see their Appendix B.9), these are 3200 10 1 10 2 n 0.0 0.2 0.4 0.6 0.8 1.0 p ( y x i ) Artifact statistics for BoolQ before and after local edits z = 2 i.i.d. edits Figure 2: Dataset artifacts in original BoolQ validation set vs. contrast set created with T ailor using (Gardner et al., 2021)'s statistical test. all a type of data recombination in Table 3, and we can unify their implementations with T ailor into the aforementioned keyword replacement in 5.1. We explore whether T ailor can be combined with noisy automated labeling for data augmentation. For the Stanford Natural Language Inference (SNLI) task (Bowman et al., 2015), we show that data augmentation with T ailor perturbations increases model robustness to inference heuristics. Min et al. (2020) find that augmenting SNLI training data by swapping hypotheses' subject / objects ( e.g., This collection contains 16 El Grecos. (cid:57) 16 El Grecos contain this collection ) improves performance on HANS, a challenge set for diagnosing fallible syntactic heuristics in NLI models (McCoy et al., 2019). Following this, we use T ailor to perturb hypotheses with the SWAP_CORE operation such that original hypothesis  premise and perturbed hypothesis  new hypothesis . We finetune R o BERT a base (Liu et al., 2019) on di  erent training datasets: original SNLI train data (unaugmented baseline), SNLI train augmented with Min et al. (2020) (augmented baseline, referred to as Syntactic Perturb. in Table 6), and SNLI train augmented with T ailor perturbations. We augment  2% of SNLI train. 12 For each subset, we train 20 models with di  erent random seeds. We evaluate each classifier on the in-domain SNLI test set and the out-of-domain HANS test set. 13 As shown in Table 6, augmentation with T ailor leads to 5.8-point gain on HANS overall, HANS and a 29.2 -point gain on non-entailment, compared to the unaugmented baseline. The improvements are significant, with t =  6 . 42, p < 10  3 12 We augment the 549,367 SNLI train instances with 10,987 new instances. significant gains on the HANS challenge set, without decreasing in-domain accuracy. using Student's t-test. Thus, T ailor perturbations decrease reliance on the lexical-overlap-based inference heuristic for NLI. Furthermore, T ailor outperforms Syntactic Perturb. , an augmented baseline designed specifically for NLI. We hypothesize that although they create augmentations through similar transformations, Min et al. (2020)'s approach is limited to inputs with specific syntactic configurations, whereas T ai lor 's SWAP_CORE argument is applicable to any AGENT and PATIENT arguments. Thus, T ailor is useful for improving model robustness  more so than template-based approaches. Controllable text generation has been widely used to influence various properties of generated text for text summarization (Peng et al., 2019), data augmentation (Lee et al., 2021), style transfer (Reid and Zhong, 2021; Madaan et al., 2020a), adversarial example generation (Iyyer et al., 2018), etc. Most generators take simple controls like tense (Hu et al., 2017), topic (Keskar et al., 2019), or sentiment polarity (Dathathri et al., 2020), which underspecify desired transformations. In contrast, T ailor concretizes otherwise sparse controls ( e.g., we can specify making a sentence more negative through negation .) Recent works incorporating syntactic structures for paraphrasing (Iyyer et al., 2018; Chen et al., 2019; Bao et al., 2019; Kumar et al., 2020; Sun et al., 2021; Huang and Chang, 2021) or discrete semantic signatures for diverse generation (Weir et al., 2020) are similar to T ailor in their high-dimensional specification. Also closely related are methods that reconstruct sentences from structured semantic representations. The most similar related work is InFillmore (Ou et al., 2021), which uses semantic representations derived from FrameNet with constrained decoding to guide generation. While InFillmore tunes the higher-level semantics of a sentence, T ailor 's semantic controls incorporate fine-grained information about the location and semantics of tex-3201 tual phrases; in addition, we demonstrate two new applications for semantically-guided generation, contrast set generation and data augmentation. Abstract Meaning Representation (Banarescu et al., 2013; Mager et al., 2020) is an alternative semantic representation worth exploring for data perturbation, as it may further enable controls on entity recursions (Damonte and Cohen, 2019), though expressing such relationships is nontrivial. Controlled generators have also been successfully used to perturb text for model training, evaluation, and explanation. They usually rely on application-specific labels (Ross et al., 2021; Madaan et al., 2020b; Sha et al., 2021; Akyrek et al., 2021) or require pairs of original and perturbed sentences (Wu et al., 2021), which are expensive to generalize. Also related are the creation of minimally edited datasets, either through manual rewriting (Gard-ner et al., 2020; Kaushik et al., 2020), or creating perturbation templates (Andreas, 2020; Li et al., 2020; Ribeiro et al., 2020; Wu et al., 2019); T ailor reduces the human e  orts these studies require. We propose T ailor , a system that enables task-agnostic, complex and context-aware perturbations. T ailor demonstrates that it is possible to drive fine-grained perturbations with semantic features directly derived from an instance. Crucially, it shows that incorporating classical linguistic structures with modern large-scale neural architectures is feasible: With the help of modern pretrained large models, PropBank-style shallow semantic representations can help steer generation towards desired meanings. Factors that a  ect T ailor 's capability. Though broadly applicable, T ailor 's controllability and effectiveness vary for di  erent inputs. First, creating automatic perturbations with T ailor requires external SRL predictors, which can be noisy on rare semantic roles or low-resource languages. 14 Empirically, this did not seem to be a bottleneck, as exposing biases in downstream tasks does not usually require rarity at the semantic role level ( e.g., testing syntactic heuristics in NLI requires swapping only agents and patients). However, perturbing more challenging linguistic phenomena may re-14 Note that while T ailor is designed to be language agnostic, we only evaluated it on English. We also notice T ailor can sometimes produce degenerate outputs. We hypothesize that this is a byproduct of unlikelihood training  i.e., the generator learns to reduce the likelihood of negative sequences by generating tokens that are very unlikely to appear in natural text. Generation hyperparame-ters ( e.g., number of beams) can reduce the number of degenerate outputs. While we perform unlikelihood training at the sequence level, future work can investigate the e  ect of penalizing generation at the level of tokens or spans, which may provide finer-grained signals for which spans should be considered unlikely, as well as more strategically balancing positive and negative samples. Extending T ailor . We believe the T ailor generator is well-suited for controlled generation tasks beyond the perturbation-based tasks we explore. Given key entities or arguments as keywords and fully masked contexts, we envision T ailor can help generate arguments (Schiller et al., 2021), compositionally augment data (Akyrek et al., 2021), or generate captions (Chen et al., 2020). In particular, as shown in 5, T ailor 's human-readable controls can support humans on data curation, which suggests that designing NLP models for augmenting human capabilities is a promising direction. The design of controls is also worthy of in-depth exploration. As mentioned in 7, AMR might be an alternative for semantic representation, if our primary goal is to express non-sequential relations. On the other hand, dependency parsing labels are useful for syntactic changes; future work may try to balance syntactic and semantic controls. Having noted these opportunities, we believe T ailor is already a powerful tool for perturbation, particularly for tasks where compositional changes are required. T ailor is opensource, and available at https://github.com/allenai/tailor . We thank Ana Marasovic, William Merrill, Thomas R. McCoy, and Daniel S. Weld for their helpful suggestions, and the anonymous reviewers for their feedback. Hao Peng is supported by a Google Fellowship.
abstain abstain method abstain method abstain objective objective abstain result abstain abstain abstain abstain abstain abstain abstain method other abstain method abstain result abstain result abstain abstain abstain abstain objective method objective abstain objective abstain method abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain method other abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain method abstain abstain method method method abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain method method abstain abstain method abstain abstain abstain method method abstain abstain abstain method abstain abstain method method other abstain other abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain method method abstain abstain method abstain abstain method abstain abstain method abstain other method abstain method method method method abstain abstain abstain method abstain method abstain abstain abstain method method abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other objective other other other other abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other other other