Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes ( e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, STYLEPTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on STYLEPTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, STYLEPTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation. At the heart of interactive AI systems lies the element of communication as a channel to convey intentions using different stylistic attributes. Research in human-AI interaction has focused on building dialog systems (Celikyilmaz et al., 2018), virtual assistants (Cooper et al., 2004), and intelligent agents (Kim et al., 2013; Liang et al., 2020a; Pittermann et al., 2010) that can communicate their intentions with specific styles for different situations, target audiences, and environments (Lample  authors contributed equally The bad service of the waitresses make me dread going sometimes. et al., 2019; Li et al., 2018). For example, expressing the same facts using either formal or informal styles can be more suitable for certain target audiences (Rao and Tetreault, 2018). What is a style in natural languages? Existing style transfer benchmarks primarily focus on individual high-level stylistic changes across sentiment (Shen et al., 2017), formality (Rao and Tetreault, 2018), politeness (Madaan et al., 2020), and writing styles (Jhamtani et al., 2017). Figure 1 provides some motivating examples to show that the high-level style transfers as commonly studied in existing benchmarks ( e.g. Yelp for sentiment (Shen et al., 2017) and GYAFC for formality (Rao and Tetreault, 2018)) can in fact be seen as composed from a dictionary of fine-grained style constructs. This alternative way of studying styles brings additional flexibility that enables fine-grained control with the possibility to compose a broader space of styles spanning tense, sentence structure, phrase emphasis, and information contained in the sentence. However, the missing link is a benchmark dataset that offers this type of fine-2117 grained style constructs, with the controllability to compose these stylistic transfers. 2 Related Work Several lines of research have aimed to formalize styles in natural languages through computational and linguistic perspectives (DiMarco and Hirst, 1993). The first systematic formulation of styles was by McDonald and Pustejovsky (1985) and later extended by DiMarco and Hirst (1993) to 4 representational categories including lexical, syntax, thematic, and semantic aspects. To fill this gap, we leverage research in linguistics to study formulations of styles across 4 representational categories: lexical, syntax, semantics, and thematics, that span the fundamental atomic transfers that text can undergo (McDonald and Pustejovsky, 1985; DiMarco and Hirst, 1993). Using these insights, we introduce a large-scale benchmark with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning the most atomic lexical, syntactic, semantic, and thematic style constructs, as well as (2) compositions of multiple transfers which model how fine-grained style constructs compose to form more complex, high-level transfers. Our dataset, called STYLEPTB, builds upon Penn Treebank (Marcus et al., 1993) by annotating each sentence undergoing these fine-grained style constructs, resulting in a large-scale resource spanning 59 , 767 sentence pairs across 21 individual styles and an additional 35 , 887 sentence pairs across 32 compositions of multiple styles. STYLEPTB allows us to study the performance of state-of-the-art style transfer models when faced with the new challenge of fine-grained style transfer. It is interesting to observe that these models, while capable of performing high-level semantic changes, struggle with fine-grained changes, particularly in the syntactic and thematic domains. A second analysis in this paper is to see how these models can handle compositions of multiple style constructs as a step towards controllable high-level style transfer. However, we find that current models have an even more difficult time composing multiple styles. As a step towards this desiderata, we also propose an approach (CS-GPT) based on pre-trained language models (Radford et al., 2019) that achieves compositional style transfer. We believe that STYLEPTB will bring novel challenges that we hope will encourage research in controllable generation, compositionality of styles, and learning disentangled representations (John et al., 2019). From a broader perspective, we conclude with the observation that controllable style transfer models trained on STYLEPTB can help mitigate social biases in pre-trained language models. Following this, there has been some early efforts applying stylistic analysis into dialog generation (Hovy, 1987), machine translation (DiMarco, 1994), and text generation (Gatt and Krahmer, 2018). We take advantage of this prior work when formalizing our new STYLEPTB dataset. Current benchmarks for style transfer focus on high-level style definitions such as transfer of sentiment (Shen et al., 2017; Lample et al., 2019; Li et al., 2018; Wu et al., 2019), politeness (Madaan et al., 2020), formality (Rao and Tetreault, 2018; Liu et al., 2020; Krishna et al., 2020), writing styles (Jhamtani et al., 2017; Syed et al., 2020; Jin et al., 2020) and some other styles (Kang and Hovy, 2019). However, these only focus on only high-level styles, unlike STYLEPTB. Computational models for style transfer span statistical NLP methods (Hovy, 1987; Xu et al., 2012), neural generative models (Prabhumoye et al., 2018; Lample et al., 2019; He et al., 2020), and Retrieve-and-Edit approaches (Li et al., 2018; Hashimoto et al., 2018; Guu et al., 2018; Sud-hakar et al., 2019; Madaan et al., 2020). These approaches work for a predefined set of styles but are unable to generalize to compositions of styles. Evaluating style transfer is difficult due to the diversity of plausible transferred sentences. In addition to automatic scores such as BLEU, perplexity, or binary classification accuracy of style transfer (Hu et al., 2017; Lample et al., 2019; He et al., 2020), other automatic metrics (Fu et al., 2018; Mir et al., 2019) and human evaluation are also commonly used (Li et al., 2018; Shen et al., 2017). As a step towards enabling fine-grained control with the possibility to compose a broader space of styles, we first define style constructs at fine-grained levels spanning lexical, syntactic, semantic, and thematic aspects. When selecting these style constructs, we have 2 goals in mind: (1) they should be representative of the four aspects (lexical, syntactic, semantic, thematic) following the formal categorizations in DiMarco and Hirst (1993), and (2) the transfers should be consistent ( i.e. well-defined such that if multiple annotators are asked to modify the same sentence, the results will be sim-ilar). With these goals in mind, we summarize the 2118 Aspect Transfer Original Sentence Additional Info/ Emphasis Transferred Sentence LEXICAL Noun synonym replacementThe shift wo n't affect operations. Semantic transfers are changes to the meaning of sentences (Bagha, 2011) that not only extend beyond lexical (Cruse et al., 1986) and syntax-level (Kratzer and Heim, 1998) changes, but also include modifications using indirect information such as referring (Strawson, 1950), situations (Barwise and Perry, 1981) or intentions and extensions (All-wood et al., 1977). As a starting point, we defined two simple types of semantic transfers: (1) Info removal: 3 transfers on different deletions: word-level (removing adjectives and adverbs), phrase 2119 435 594 735 928 1051 1297 1247 1432 0 200 400 600 800 1000 1200 1400 1600 5 6 7 8 9 10 11 12 Adjectives8% Nouns31% Verbs19% Determiners10% Adverbs6% Others26% 500 417408403 308 257241203197176174173145145138137135123119113113105 94 91 90 90 89 88 86 85 0 100 200 300 400 500 600 m r . following 21 chosen fine-grained style constructs spanning 4 categories and also provide detailed examples in Table 1. Lexical transfers are those at fine-grained lexicon levels ( i.e. vocabulary or words) that include word constitutions (Heine et al., 2002) and word meaning (Cruse et al., 1986). As a starting point, we selected two types of lexical transfers: synonym/antonym replacements ( 6 transfers that replace nouns/verbs/adjectives with their synonyms/antonyms), and frequency-based replacements ( 2 transfers that replace words with their most/least appeared synonyms). The syn-onym/antonym resources are taken from Wordnet (Fellbaum, 2012). sentences (Chomsky, 2002) without affecting the content (Akmajian and Heny, 1980). We selected three simple syntax transfers: tense changes ( 3 transfers: to past/present/future tense), voice changes ( 2 transfers: active to/from passive), proposition position changes ( 2 transfers: front to/from back). level (removing propositions), and substatement level (removing entire substatements) that represent referring and situations , as well as (2) Info addition: 1 transformation that adds a given piece of information regarding a particular phrase in the current sentence representing extension . Thematic transfers concern the placing of emphasis across different parts in a sentence (Steven-son et al., 1994) to highlight different aspects of the same event (DiMarco, 1994). We defined two emphatic transfers across adjectives and verbs (ac-tions). As an example of adjective emphasis, the hot meat is on the table emphasizes location, while the meat on the table is hot emphasizes the hot temperature. To enforce consistency across annotators, we require adjective emphasis to rewrite the sentence into a be-statement of the emphasized adjective (as in the example above). Analysis: To evaluate how useful these 21 selected atomic transfers are, we randomly sampled 50 sentence pairs from GYAFC and 50 sentences from Yelp with their reference transfer generated by Deep Latent Sequence Model (He et al., 2020) and manually tried to complete the transfers by composing one or more of the 21 atomic transfers we have defined, together with capitalization fixes and word-spelling fixes. We found that 72% of transfers from GYAFC, and 82% of transfers from Yelp can be done this way. Specifically, in GYAFC, 24% require one atomic transfer, and another 48% require composing multiple atomic transfers; in Yelp, 52% require one or less atomic transfers and another 30% require composing multiple atomic transfers. The results of this analysis suggest that STYLE PTB's dictionary of atomic styles is already a good start in studying compositional style transfer. STYLE PTBatomic transfers and their composition do indeed span a large percentage of current high-level style transfers. Using these selected 21 style constructs, we now illustrate the steps towards collecting and annotating parallel sentences across style transfers. We use Penn Treebank (PTB) (Marcus et al., 1993) as our source of sentences. Additionally, the availability of parse trees in PTB allows us to automate the majority of syntactic transfers using rule-based methods. We begin with a total of 43 , 948 sentences in the full PTB before removing sentences that are incomplete, too long (over 12 words), or too short (less than 5 words). This leaves 7 , 719 sentences (see Figure 2 for statistics and Appendix A.1 for full details). Automated rule-based transfers: For 18 of the 21 transfers (lexical, syntax, and semantic transfers except Info Addition), we defined rule-based transfers using NLTK (Loper and Bird, 2002), parse trees (syntax, semantics), and WordNet (lexical). After human quality control, the total number of sentences transferred is listed in Table 2 (see Appendix A.2 for more details on automated generation and Appendix A.4 for human evaluation on quality of generated sentences) Transfers with human annotations: For the remaining 3 transfers, we have human annotators (via Amazon Mechanical Turk) manually rewrite them due to the difficulty of automating the process. See Appendix A.3 for details on the data generation, human annotation and quality assurance process for each of the three transfers. After annotations and quality control, we obtained 696 rewritten sentences for adjective emphasis, 1201 rewritten sentences for verb emphasis, and 2114 2120 Category Transfer Number Lexical Noun synonym replacement 5948 Noun antonym replacement 2227 Verb synonym replacement 2574 Verb antonym replacement 1284 ADJ synonym replacement 434 ADJ antonym replacement 1146 Most frequent synonym replacement 4722 Least frequent synonym replacement 7112 Syntax To future tense 7272 To present tense 4365 To past tense 4422 Active  passive 2808 PP front  back 467 Semantics(informationdeletion) ADJ or ADV removal 4863 PP removal 4767 Substatement removal 1345 Information Addition 2114 Thematic Verb/action emphasis 1201 Adj emphasis 696 Table 2: STYLEPTB is a large-scale resource spanning 59 , 767 sentence pairs across 21 individual styles. valid sentence-information pairs with their transferred sentence with information added. Lexical transfers can be done by replacing individual words and is simple to evaluate. To evaluate the difficultly of the remaining 13 syntax, semantic, and thematic transfers, we calculated the token-level ( i.e. word level) Hamming distance between original and transferred sentences. Using this metric, we categorized these 13 transfers into easy, medium and hard categories (see Table 3). We also evaluated semantic measures from BERT embeddings (Devlin et al., 2018) but found it less correlated with human judgment (see Appendix A.5). To allow for compositionality, we also generated compositional data that includes parallel pairs of sentences linked by multiple sequential transfers. To compose automatic transfers, we applied a sequence of rule-based transfers starting with parse trees (see Table 4). To compose transfers that involve human annotations, we apply a sequence of reverse changes on the original sentences with parse trees (since human rewritten sentences no longer have parse trees), before chaining the sequence of automatic reverse transfers with the final human-annotated transfer (see Figure 3). We extend the pre-trained GPT2 language model (Radford et al., 2019) for parallel style transfer by giving it designated style transfer tokens as input in addition to the source sentence. For example, for each individual binary style s i , we define a style transfer token  i  { 0 , 1 , 2 } where  i = 0 represents keeping s i unchanged,  i = 1 represents a change from s i = 0 to s i = 1 , and vice versa for  i = 2 . We likewise extend the definition of  i for styles taking more than 2 values. Given a parallel (source, target) pair ( s, t ) , we define the appropriate transfer token   { 0 , 1 , 2 } 2121 Easy Transfers Baseline Model BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE_L CiDER To Future Tense GPT2 0.895 0.852 0.813 0.778 0.540 0.899 7.709 SEQ 2 SEQ 0.527 0.368 0.261 0.188 0.173 0.531 1.525 RETRIEVEEDIT 0.899 0.854 0.815 0.778 0.531 0.901 7.731 HUMAN 0.954 0.915 0.884 0.855 0.636 0.964 9.174 ADJ or ADV Removal GPT2 0.647 0.508 0.394 0.308 0.313 0.652 3.259 SEQ 2 SEQ 0.450 0.274 0.172 0.112 0.140 0.469 1.171 RETRIEVEEDIT 0.897 0.841 0.786 0.731 0.511 0.919 7.461 HUMAN 0.933 0.894 0.870 0.847 0.591 0.965 8.924 Medium Transfers Baseline Model BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE_L CiDER Substatement Removal GPT2 0.430 0.332 0.247 0.176 0.250 0.588 3.090 SEQ 2 SEQ 0.317 0.192 0.110 0.001 0.100 0.368 1.041 RETRIEVEEDIT 0.706 0.678 0.647 0.607 0.405 0.767 6.183 HUMAN 0.731 0.720 0.705 0.685 0.607 0.788 7.691 Information Addition GPT2 0.479 0.305 0.189 0.121 0.207 0.475 1.359 SEQ 2 SEQ 0.345 0.180 0.094 0.053 0.098 0.335 0.632 RETRIEVEEDIT 0.493 0.396 0.328 0.275 0.284 0.603 3.401 HUMAN 0.846 0.762 0.690 0.624 0.521 0.892 6.863 Hard Transfers Baseline Model BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE_L CiDER Active To Passive GPT2 0.476 0.329 0.238 0.189 0.216 0.464 1.820 SEQ 2 SEQ 0.373 0.220 0.141 0.103 0.131 0.345 0.845 RETRIEVEEDIT 0.681 0.598 0.503 0.427 0.383 0.663 4.535 HUMAN 0.931 0.881 0.835 0.795 0.587 0.905 8.603 Adjective Emphasis GPT2 0.263 0.079 0.028 0.000 0.112 0.188 0.386 SEQ 2 SEQ 0.187 0.058 0.018 0.000 0.059 0.179 0.141 RETRIEVEEDIT 0.387 0.276 0.211 0.164 0.193 0.369 1.679 HUMAN 0.834 0.753 0.679 0.611 0.522 0.811 6.796 Verb/Action Emphasis GPT2 0.309 0.170 0.095 0.041 0.140 0.292 0.593 SEQ 2 SEQ 0.289 0.127 0.066 0.038 0.098 0.275 0.300 RETRIEVEEDIT 0.416 0.284 0.209 0.148 0.223 0.423 1.778 HUMAN 0.649 0.569 0.493 0.421 0.433 0.693 5.668 Table 5: Evaluation results on easy (top), medium (middle), and hard (bottom) transfers. 6 Experiments We test the performance of current style transfer models on STYLEPTB. Anonymized data and code is included in the supplementary, and we present extra details and results in Appendix B and C. 2122 6.1 Datasets and Metrics We use STYLEPTB and evaluate on the 13 non-lexical transfers (since lexical changes works best with fixed word substitutions). and train using maximum likelihood estimation to predict every word t j , for j = 1 , 2 , . . . , T , in the target sentence given the source and  :   = arg max  E ( s,t )D  T  j = 1 log p  ( t j ; s,  )  , (1) where  denotes the pre-trained GPT2 parameters and   denotes the parameters after fine-tuning on STYLEPTB. Note that we also train the model to reconstruct the same source sentence again when setting  = 0 (no style change), which we found to help bridge the domain shift between data used to pre-train GPT2 and sentences in STYLEPTB. As a step towards compositionality , we also train with (source, target) pairs that undergo multiple atomic style transfers as provided in STYLEPTB, resulting in multiple style transfer tokens  i being activated at the same time. We call the resulting model CS-GPT (Compositional Style GPT) and show its architecture in Figure 4. Learning separate representations for each  i results in disentangled style variables that can then be composed as desired. Another benefit of using disentangled style variables is the ability of a single model in performing multiple style transfers. 6.3 Results and Observations We evaluate these 3 baseline models on the style transfers in STYLEPTB and show results in Table 5. We make the following observations: Baseline comparisons: RETRIEVEEDIT performed equally well compared to GPT2 in some transfers such as To Future Tense and performs significantly better compared to GPT2 in most transfers. When qualitatively observing the generated sentences, we found that while GPT2 can learn syntactic and semantic transfers, they suffer in reconstructing the rest of the sentence ( e.g. making word repetitions). This was not an issue Clarity Context Style GPT2 1 . Please refer to Appendix B.1 for dataset preprocessing details. Automated evaluation metrics consists of automatic BLEU scores, METEOR scores, ROUGE_L scores, and CiDER scores between generated and ground truth sentences (Sharma et al., 2017). In addition, we did human evaluations on random sets of 10 samples generated by each model for each transfer. We followed prior work (He et al., 2020) and had 2 independent annotators each rate transferred sentences on three aspects (clarity/grammar, content preservation, style change) on a 1  5 Likert scale, and takes average. We evaluate the following baselines commonly used in style transfer. Since none of these existing models handle compositions of styles, we train separate models on each of the 13 transfers. 1) GPT2: We fine-tune pre-trained GPT2 (Rad-ford et al., 2019) on each transfer with the source as input and predicting the target using MLE, similar to Liu et al. (2020); Syed et al. (2020). 2) SEQ 2 SEQ : A Seq2Seq model (Sutskever et al., 2014) with attention trained using MLE (Zhou et al., 2020; Jin et al., 2020). 3) RETRIEVEEDIT : Given input x , a retriever is trained to pick a similar training example ( x  , y  ) . We treat y  as our prototype and use a trained editor to edit it into desired output y (Guu et al., 2018; Madaan et al., 2020). 4) HUMAN : We also report human performance for each style transfer by having two independent human annotators manually perform the style transfer on 20 sampled sentences. for RETRIEVEEDIT since it works by editing the sentence from the prototype. Both GPT2 and RETRIEVEEDIT significantly outperform SEQ 2 SEQ models on all 13 non-lexical transfers. Difficulties of transfers: We also compare the relative difficulty of transfers based on the automatic metrics described in Section 4.3. In line with our Hamming distance metric, we found that thematic transfers are especially difficult all three baselines struggled on this task, which is intuitive because shifting emphasis requires completely different sentence structure changes on sentences and emphasized words. We found that GPT2 and SEQ 2 SEQ tend to struggle with grammar and word repetitions, while RETRIEVEEDIT sometimes follows the structural edits in the chosen (and often completely unfitting) examples, resulting in malformed outputs (see examples in Appendix C.1). All current methods significantly fall short of human performance especially on hard transfers. Therefore, we believe that STYLEPTB brings novel challenges that will spark future research in modeling fine-grained style changes. Human evaluation: We sampled 10 transferred sentences from each automatic generations models for each transfer and asked 2 independent annotators to rate them. We show average results below for one of the hard transfers (Verb Emphasis). From Table 6, we found that all approaches fall far short of human performance, which was judged by a separate human as having almost perfect clarity, content, and style metrics. Furthermore, GPT2 gets higher style scores while RETRIEVEEDIT excels at grammar and content preservation, which further supports our qualitative observations above. Full results for human evaluations are available in Table 17 in Appendix C.1. 2123 Transfers Model BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE_L CiDER ToFuture+ActiveToPassive GPT2 0.391 0.222 0.120 0.065 0.167 0.373 0.866 CS-GPT-ZERO 0.419 0.243 0.114 0.047 0.209 0.325 1.238 CS-GPT 0.496 0.340 0.240 0.185 0.217 0.479 1.800 ToPast+PPRemoval GPT2 0.714 0.640 0.573 0.510 0.374 0.724 5.152 CS-GPT-ZERO 0.542 0.389 0.268 0.182 0.314 0.535 2.103 CS-GPT 0.772 0.695 0.624 0.564 0.421 0.775 5.585 Table 7: Results on compositions of transfers: CS-GPT with compositional data works better than CS-GPT-ZERO (without compositional data), and sequentially applying GPT2 models. 7 Broader Impact: Mitigating Biases Unconditional language models have been shown to perpetuate undesirable stereotypes during generation which disproportionately harm underrepresented social groups (Liang et al., 2020b; Ravfogel et al., 2020; Sheng et al., 2020). As one possible application of fine-grained style transfer (in addition to many others), we hypothesize that more fine-grained control over the generated outputs can 2124 Transfer To Future + Passive To Active To Past + PP Removal Source Sentence NUM % was risen by sales to NUM billion from NUM billion. As a step towards learning compositional transfers, we implemented the following baselines: 1. GPT2: Sequentially applying the GPT2 model trained for single transfers multiple times to perform compositional transfers. 2. CS-GPT: Our proposed CS-GPT model (detailed in Section 5) trained on compositional transfer pairs found in STYLEPTB. 3. CS-GPT-ZERO : An ablation of CS-GPT trained only on individual style changes but tested in a zero-shot setting on compositional transfers. We evaluated these models on two compositional transfers: Tense+Voice (composing tense changes and active/passive voice changes), and Tense+PP Removal (composing tense changes and PP Re-moval). We conveniently used the numerical pre-fixes in the datasets as transfer tokens. The results are shown in Table 7 and we make the following observations: CS-GPT works best for compositional transfers: CS-GPT significantly outperforms existing methods for compositional style transfer. This is expected, as CS-GPT is trained on the full compositional dataset, while CS-GPT-ZERO is only trained on part of the compositional data and SEQGPT is trained on single-transfer parallel data. Qualitatively, we observed that CS-GPT is able to perform each required transfer at the same time, producing outputs with relatively low reconstruction error compared to the other two methods. We included a few samples generated by the three models in Table 9 with more examples in Appendix C.2. whether CS-GPT can learn to compose transfers in a zero-shot manner. While CS-GPT outperforms CS-GPT-ZERO and existing models, all still struggle to perform zero-shot compositions. We noticed that CS-GPT-ZERO usually only performs one of the necessary transfers: e.g. in a Tense+Voice task, CS-GPT-ZERO tends to only make the tense change, not the voice change. Quantitatively, in the Tense+PP Removal dataset, CS-GPT-ZERO performs much worse than either CS-GPT or sequentially applying GPT2; in Tense+Voice dataset, CS-GPT-ZERO is similar to GPT2. We believe that sequentially applying GPT2 accumulates errors present in each one. Training on compositional styles may improve fine-grained styles: We observe that CS-GPT trained on compositional data can achieve similar performance as compared to GPT2 trained specifically for that one single transfer. Sometimes CS-GPT even outperforms GPT2, as shown in Table 8 (see Table 20 in Appendix for full results). Therefore, CS-GPT has leveraged compositional structure and data to perform strongly on multiple single and compositional transfers with just one model. help to control the output sentence and mitigate bias. To validate our hypothesis, we perform a proof-of-concept experiment: we show clearly biased sentences GPT2 generated via given prompts from Sheng et al. (2019) (shown underlined), before rewriting them using fine-grained transfers learned by our model. 1. Simple lexical transfers can mitigate bias by replacing certain stereotyped nouns with alternatives (through synonym/antonym replacement): GPT2: The Black man drove a car to a house where the victim had a family member. Antonym replacement: The Black man drove a car to a house where the beneficiary had a family member. Another technique that may be useful is to replace the key nouns with vague pronouns: GPT2: The Hispanic man said that he was walking his dog when the suspect yelled. Replace nouns: The hispanic man said that he was walking his dog when they yelled. In the following two examples, the information removal is done by GPT2: GPT2: The woman walks home from the theatre crying . Info removal: The woman walks home from the theatre. GPT2: The Asian man drove a black BMW to the station to meet with an officer who would be taking him to a jail . Info removal: The Asian man drove a black BMW to meet with an officer. We can also replace the removed part with uniformly sampled information with Info Addition. Since existing baselines currently struggle, we present a manual example as proof of concept: GPT2: The woman worked as a medical assistant in a health center. 2. Semantic changes can also help by removing biased information from generated test using information removal. Info removal: The woman worked in a health center. Info addition with worked as doctor: The woman worked as a doctor in a health center. Finally, we performed a qualitative evaluation: we sampled 49 sentence completions using prompts from Sheng et al. (2019) that focus on stereotypical associations between gender and occupations (e.g. The woman worked as a babysitter. ). We obtained post-processed versions using Info removal followed by Info addition with uniformly sampled new occupations. When presented to two independent human annotators, they judged 22 / 49 sentences as showing significantly lower bias with the remaining showing little or no bias change, indicating that fine-grained style transfer presents a new perspective to mitigating social biases in language models (see Appendix D for evaluation details). In this paper, we propose a large-scale benchmark, STYLEPTB, for fine-grained style transfer spanning atomic lexical, syntactic, semantic, and thematic changes as well as their compositions into high-level transfers. We show that STYLEPTB provides an important step towards training more controllable text generators and removing social biases from generated text. However, existing style transfer models struggle to perform fine-grained changes and have an even more difficult time composing multiple styles. As a result, STYLEPTB brings novel challenges that we hope will inspire future research in controllable text generation, compositional models, and style disentanglement. PPL and LM were supported in part by the National Science Foundation (Awards #1750439, #1722822) and National Institutes of Health. HP and BP are supported by the DARPA D3M Program and The Boeing Company. RS was supported in part by NSF IIS1763562 and ONR Grant N000141812861. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation, National Institutes of Health, DARPA, The Boeing Company, or the ONR and no official endorsement should be inferred. We would also like to acknowledge NVIDIA's GPU support and the anonymous reviewers for their constructive comments.
abstain abstain method result objective abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method result objective abstain result result abstain abstain method abstain objective other abstain other abstain abstain other abstain other abstain other method abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method method method method abstain method abstain abstain other abstain abstain method abstain abstain method abstain abstain abstain method abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain objective result abstain objective other other other other other