Style transfer is the task of rewriting a sentence into a target style while approximately preserving content. While most prior literature assumes access to a large style-labelled corpus, recent work (Riley et al., 2021) has attempted few-shot style transfer using just 3-10 sentences at inference for style extraction. In this work, we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim . We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases. When compared to prior work, our model achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages. Moreover, our method is better at controlling the style transfer magnitude using an input scalar knob. We report promising qualitative results for several attribute transfer tasks (sentiment transfer, simplification, gender neutralization, text anonymiza-tion) all without retraining the model . Finally, we find model evaluation to be difficult due to the lack of datasets and metrics for many languages. To facilitate future research we crowdsource formality annotations for 4000 sentence pairs in four Indic languages, and use this data to design our automatic evaluations. 1 1 Introduction Style transfer is a natural language generation task in which input sentences need to be re-written into a target style, while preserving semantics. It has many applications such as writing assistance (Hei-dorn, 2000), controlling generation for attributes 1 Please visit the project page for the paper resources: https://martiansideofthemoon.github.io/2022/03/03/acl22.html . like simplicity, formality or persuasion (Xu et al., 2015; Smith et al., 2020; Niu and Carpuat, 2020), data augmentation (Xie et al., 2020; Lee et al., 2021), and author obfuscation (Shetty et al., 2018). Most prior work either assumes access to supervised data with parallel sentences between the two styles (Jhamtani et al., 2017), or access to a large corpus of unpaired sentences with style labels (Prabhumoye et al., 2018; Subramanian et al., 2019). Models built are style-specific and cannot generalize to new styles during inference, which is needed for applications like real-time adaptation to a user's style in a dialog or writing application. Moreover, access to a large unpaired corpus with style labels is a strong assumption. Most standard unpaired style transfer datasets have been carefully curated (Shen et al., 2017) or were originally parallel (Xu et al., 2012; Rao and Tetreault, 2018). This is especially relevant in settings outside En-7439 glish, where NLP tools and labelled datasets are largely underdeveloped (Joshi et al., 2020). In this work, we take the first steps studying style transfer in seven languages 2 with nearly 1.5 billion speakers in total. Since no training data exists for these languages, we analyzed the current state-of-the-art in few-shot multilingual style transfer, the Universal Rewriter ( UR ) from Garcia et al. (2021). Unfortunately, we find it often copies the inputs verbatim (Section 3.1), without changing their style . We propose a simple inference-time trick of style-controlled translation through English, which improves the UR output diversity (Section 4.1). To further boost performance we propose DIFFUR , 3 a novel algorithm using the recent finding that paraphrasing leads to stylistic changes (Krishna et al., 2020). DIFFUR extracts edit vectors from paraphrase pairs, which are used to condition and train the model (Figure 2). On formality transfer and code-mixing addition, our best performing DIFFUR variant significantly outperforms UR across all languages (by 2-3x) using automatic & human evaluation. Besides better rewriting, our system is better able to control the style transfer magnitude (Figure 1). A scalar knob (  ) can be adjusted to make the output text reflect the target style (pro-vided by exemplars) more or less. We also observe promising qualitative results in several attribute transfer directions (Section 6.2) including sentiment transfer, simplification, gender neutralization and text anonymization, all without retraining the model and using just 3-10 examples at inference. Finally, we found it hard to precisely evaluate models due to the lack of evaluation datasets and style classifiers (often used as metrics) for many languages. To facilitate further research in Indic formality transfer, we crowdsource formality annotations for 4000 sentence pairs in four Indic languages (Section 5.1), and use this dataset to design the automatic evaluation suite (Section 5). In summary, our contributions provide an end-to-end recipe for developing and evaluating style transfer models and evaluation in a low-resource setting. Few-shot methods are a recent development in English style transfer, with prior work using variational autoencoders (Xu et al., 2020), or prompting large pretrained language models at inference (Reif et al., 2021). Most related is the state-of-the-art TextSETTR model from Riley et al. (2021), who use a neural style encoder to map exemplar sentences to a vector used to guide generation. To train this encoder, they use the idea that adjacent sentences in a document have a similar style. Recently, the Universal Rewriter (Garcia et al., 2021) extended TextSETTR to 101 languages, developing a joint model for translation, few-shot style transfer and stylized translation. This model is the only prior few-shot system we found outside English, and our main baseline. We discuss its shortcomings in Section 3.1, and propose fixes in Section 4. Multilingual style transfer is mostly unexplored in prior work: a 35 paper survey by Briakou et al. (2021b) found only one work in Chinese, Russian, Latvian, Estonian, French. They further introduced XFORMAL, the first formality transfer evaluation dataset in French, Brazilian Portugese and Italian. 4 To the best of our knowledge, we are the first to study style transfer for the languages we consider. More related work from Hindi linguistics and on style transfer control is provided in Appendix B. 3 The Universal Rewriter ( UR ) model We will start by discussing the Universal Rewriter ( UR ) model from Garcia et al. (2021), upon which our proposed DIFFUR model is built. At a high level, the UR model extracts a style vector s from an exemplar sentence e , which reflects the desired target style. This style vector is used to style transfer an input sentence x . Concretely, consider f enc , f dec to be encoder & decoder Transformers initialized with mT5 (Xue et al., 2021b), which are composed to form the model f ur . The UR model extracts the style vector using the encoder representation of a special [CLS] token prepended to e , and adds it to the input x representations for style transfer, f style ( e ) = s = f enc ( [CLS]  e )[0] f ur ( x, s ) = f dec ( f enc ( x ) + s ) where  is string concatenation, + vector addition. Learning Style Transfer by Exemplar-driven Denoising : To learn a style extractor, the Universal Rewriter uses the idea that two non-overlapping spans of text in the same document are likely to have the same style. Concretely, let x 1 and x 2 be 4 We do not use this data since it does not cover Indian languages, and due to Yahoo! L6 corpus restrictions for industry researchers (confirmed via author correspondence). two non-overlapping spans. Style extracted from one span ( x 1 ) is used to denoise the other ( x 2 ),  x 2 = f ur ( noise ( x 2 ) , f style ( x 1 )) L denoise = LCE ( x 2 , x 2 ) where LCE is the standard next-word prediction cross entropy loss function and noise(  ) refers to 20-60% random token dropping and token replacement. This objective is used on the mC4 dataset (Xue et al., 2021b) with 101 languages. To build a general-purpose rewriter which can do translation as well as style transfer, the model is additionally trained on two objectives : (1) supervised machine translation using the OPUS-100 parallel dataset (Zhang et al., 2020), and (2) a self-supervised objective to learn effective style-controlled translation; more details in Appendix C. During inference (Figure 1), consider an input sentence x and a transformation from style A to B (say informal to formal ). Let SA , SB to be exemplar sentences in each of the styles (typically 3-10 sentences). The output y is computed as, s A = 1 | SA | (cid:88) y  SA f style ( y ) s B = 1 | SB | (cid:88) y  SB f style ( y ) y = f ur ( x,  ( s B  s A )) where  acts as a control knob to determine the magnitude of style transfer, and the vector subtraction helps remove confounding style information. 5 3.1 Shortcomings of the Universal Rewriter We experimented with the UR model on Hindi formality transfer, and noticed poor performance. We noticed that UR has a strong tendency to copy sentences verbatim  45.5% outputs were copied exactly from the input (and hence not style transferred) for the best performing value of  . The copying increase for smaller  , making magnitude control harder. We identify the following issues: 1. Random token noise leads to unnatural inputs & transformations: The Universal Rewriter uses 20-60% uniformly random token dropping or replacement to noise inputs, which leads to ungrammatical inputs during training. We hypothesize models tend to learn grammatical error correction, which encourages verbatim copying during 5 Garcia et al. (2021) also recommend adding the style vectors from the input sentence x , but we found this increased the amount of verbatim copying and led to poor performance. inference where fluent inputs are used and no error correction is needed. Moreover, token-level noise does not differentiate between content or function words, and cannot do syntactic changes like content reordering (Goyal and Durrett, 2020). Too much noise could distort semantics and encourage hallucination, whereas too little will encourage copying. 2. Style vectors may not capture the precise style transformation: The Universal Rewriter extracts the style vector from a single sentence during training, which is a mismatch from the inference where a difference between vectors is taken. Without taking vector differences at inference, we observe semantic preservation and overall performance of the UR model is much lower. 6 3. mC4 is noisy : On reading training data samples, we noticed noisy samples with severe language identification errors in the Hindi subset of mC4. This has also been observed recently in Kreutzer et al. (2022), who audit 100 sentences in each language, and report 50% sentences in Marathi and 20% sentences in Hindi have the wrong language. 4. No translation data for several languages: We notice worse performance for languages which did not get parallel translation data (for the translation objective in Section 3). In Table 1 we see UR gets a score 7 of 30.4 for Hindi and Bengali, languages for which it got translation data. However, the scores are lower for Kannada, Telugu & Gujarati (25.5, 22.8, 23.7), for which no translation data was used. We hypothesize translation data encourages learning language-agnostic semantic representations needed for translation from the given language, which in-turn improves style transfer. While the Universal Rewriter model has a strong tendency to exactly copy input sentences while rewriting sentences in the same language (Sec-tion 3.1), we found it is an effective style-controlled translation system. This motivates a simple inference-time trick to improve model outputs and reduce copying  translate sentences to English ( en ) in a style-agnostic manner with a zero style 6 This difference possibly helps remove confounding information (like semantic properties, other styles) and focus on the specific style transformation. Since two spans in the same document will share aspects like article topic / subject along with style, we expect these semantic properties will confound the style vector space obtained after the UR training. 7 Using the rAGG style transfer metric from Section 5.5. vector 0 , and translate back into the source language ( lx ) with stylistic control. where x is the input sentence, s A , s B are the styles vectors we want to transfer between, en, lx are language codes prepended to indicate the output language (Appendix C). Prior work has shown that backtranslation is effective for paraphrasing (Wieting and Gimpel, 2018; Iyyer et al., 2018) and style transfer (Prabhumoye et al., 2018). While style-controlled backtranslation is an effective strategy, it needs two translation steps. This is 2x slower than UR , and semantic errors increase with successive translations. To learn effective style transfer systems needing only a single generation step we develop DIFFUR , a new few-shot style transfer training objective (overview in Figure 2). DIFFUR tackles the issues discussed in Section 3.1 using paraphrases and style vector differences. Paraphrases as a noise function : Instead of using random token-level noise (Issue #1 in Section 3.1), we paraphrase sentences to noise them during training. Paraphrasing modifies the lexical & syntactic properties of sentences, while preserving fluency and input semantics. Prior work (Kr-ishna et al., 2020) has shown that paraphrasing leads to stylistic changes, and denoising can be considered a style re-insertion process. To create paraphrases, we backtranslate sentences from the UR model 8 with no style control (zero vectors used as style vectors). To increase diversity, we use random sampling in both translation steps, pooling generations obtained using temperature values [0 . 4 , 0 . 6 , 0 . 8 , 1 . 0] . Finally, we discard paraphrase pairs from the training data where the semantic similarity score 9 is outside the range [0 . 7 , 0 . 98] . This removes backtransation errors (score < 0.7), and exact copies (score > 0.98). In Appendix K we confirm that our backtranslated paraphrases are lexically diverse from the input. Using style vector differences for control : To fix the training / inference mismatch for style extraction (Issue #2 in Section 3.1), we propose using style vector differences between the output and input as the stylistic control. Concretely, let x be an input sentence and x para its paraphrase. s diff = f style ( x )  f style ( x para )  x = f ur ( x para , stop-grad ( s diff )) L = LCE ( x, x ) where stop-grad(  ) stops gradient flow through s diff , preventing the model from learning to copy x exactly. To ensure f style extracts meaningful style representations, we fine-tune a trained UR model. Vector differences have many advantages, 8 Specifically, an Indic variant of the UR model is used, described in Section 4.3. Note it is not necessary to use UR for backtranslation, any good translation model can be used. 9 Calculated using LaBSE, discussed in Section 5.3. and its paraphrase removes confounding features (like semantics) present in the vectors. 2. The vector difference focuses on the precise transformation that is needed to reconstruct the input from its paraphrase. 3. The length of s diff acts as a proxy for the amount of style transfer, which is controlled using  during inference (Section 3). DIFFUR is related to neural editor models (Guu et al., 2018; He et al., 2020), where language models are decomposed into a probabilistic space of edit vectors over prototype sentences. We justify the DIFFUR design with ablations in Appendix G.1. To address the issue of no translation data (Issue #4 in Section 3.1), we train Indic variants of our models. We replace the OPUS translation data used for training the Universal Rewriter (Section 3) with Samanantar (Ramesh et al., 2021), which is the largest publicly available parallel translation corpus for 11 Indic languages. We call these variants UR-INDIC and DIFFUR-INDIC . This process significantly up-samples the parallel data seen between English / Indic languages, and gives us better performance (Table 1) and lower copy rates, especially for languages with no OPUS translation data. One issue with our DIFFUR-INDIC setup is usage of a stop-grad(  ) to avoid verbatim copying from the input. This prevents gradient flow into the style extractor f style , and as we see in Appendix H, a degradation of the style vector space. To prevent this we simply multi-task between the exemplar-driven denoising UR objective (Section 3) and the DIFFUR objective. We initialize the model with the UR-INDIC checkpoint, and fine-tune it on these two losses together, giving each loss equal weight. Automatic evaluation of style transfer is challenging (Pang, 2019; Mir et al., 2019; Tikhonov et al., 2019), and the lack of resources (such as evaluation datasets, style classifiers) make evaluation trickier for Indic languages. To tackle this issue, we first collect a small dataset of formality and semantic similarity annotations in four Indic languages (Section 5.1). We use this dataset to guide the design of an evaluation suite (Section 5.2-5.6). Since automatic metrics in generation are imperfect (Celikyilmaz et al., 2020), we complement our results with human evaluation (Section 5.7). Since no public datasets exist for formality transfer in Indic languages, it is hard to measure the extent to which automatic metrics (such as style classifiers) are effective. To tackle this issue, we build a dataset of 1000 sentence pairs in each of four Indic languages (Hindi, Bengali, Kannada, Telugu) with formality and semantic similarity annotations. We first style transfer held-out Samanantar sentences using our UR-INDIC + BT model (Sec-tion 4.1, 4.3) to create sentence pairs with different formality. We then asked three crowdworkers to 1) label the more formal sentence in each pair; 2) rate semantic similarity on a 3-point scale. Our crowdsourcing is conducted on Task Mate, 10 where we hired native speakers from India with at least a high school education and 90% approval rating on the platform. To ensure crowdworkers understood formality, we provided instructions following advice from professional Indian linguists, and asked two qualification questions in their native language. More details (agreement, compensation, instructions) are provided in Appendix E.4. Our first metric checks whether the output sentence reflects the target style. This is measured by an external classifier's predictions on system outputs. We use two variants of transfer accuracy: (1) Relative Accuracy (rACC ): does the target style classifier score the output sentence higher than the input sentence? (2) Absolute Accuracy (aACC ): does the classifier score the output higher than 0.5? Building multilingual classifiers : Unfortunately, no large style classification datasets exist for most languages, preventing us from building classifiers from scratch. We resort to zero-shot cross lingual transfer techniques (Conneau and Lample, 2019), where large multilingual pretrained models are first fine-tuned on English classification data, and then applied to other languages at inference. We experiment with three such techniques, and find MAD-X classifiers with language adapters (Pfeiffer et al., 2020b) have the highest accuracy of 81% on our Hindi data from Section 5.1. However, MAD-X classifiers were only available for Hindi, so we use 10 https://taskmate.google.com 7443 the next best XLM RoBERTa-base (Conneau et al., 2020) for other languages, which has 75%-82% accuracy on annotated data; details in Appendix E.1. Our second evaluation criteria is semantic similarity between the input and output. Following recent recommendations (Marie et al., 2021; Krishna et al., 2020), we avoid n -gram overlap metrics like BLEU (Papineni et al., 2002). Instead, we use LaBSE (Feng et al., 2020), a language-agnostic semantic similarity model based on multilingual BERT (Devlin et al., 2019). LaBSE supports 109 languages, and is the only similarity model we found supporting all the Indic languages in this work. We also observed LaBSE had greater correlation with our annotated data (Section 5.1) compared to alternatives; details in Appendix E.2. Qualitatively, we found that sentence pairs with LaBSE scores lower than 0.6 were almost never paraphrases. To avoid rewarding partial credit for low LaBSE scores, we use a hard threshold 11 ( L = 0 . 75 ) to determine whether pairs are paraphrases, SIM ( x, y (cid:48) ) = 1 if (cid:8) LaBSE ( x, y (cid:48) ) > L (cid:9) else 0 5.4 Other Metrics ( LANG , COPY , 1-g) Additionally, we measure whether the input and output sentences are in the same language ( LANG ), the fraction of outputs copied verbatim from the input ( COPY ), and the 1-gram overlap between input / output (1-g). High LANG and low COPY / 1-g (more diversity) is better; details in Appendix E.6. To get a sense of overall system performance, we combine individual metrics into one score. Similar to Krishna et al. (2020) we aggregate metrics as, AGG ( x, y (cid:48) ) = ACC ( x, y (cid:48) )  SIM ( x, y (cid:48) )  LANG ( y (cid:48) ) AGG ( D ) = 1 |D| (cid:88) x,y (cid:48) D AGG ( x, y (cid:48) ) Where ( x, y (cid:48) ) are input-output pairs, and D is the test corpus. Since each of our individual metrics can only take values 0 or 1 at an instance level, our aggregation acts like a Boolean AND operation. In other words, we are measuring the fraction of outputs which simultaneously transfer style, have 11 Roughly 73% pairs annotated as paraphrases (from dataset in Section 5.1) had L > 0 . a semantic similarity of at least L (our threshold in Section 5.3), and have the same language as the input. Depending on the variant of ACC (relative / absolute), we can derive rAGG / aAGG . An ideal system should not only be able to style transfer sentences, but also control the magnitude of style transfer using the scalar input  . To evaluate this, for every system we first determine a  max value and let [0 ,  max ] be the range of control values. While in our setup  is an unbounded scalar, we noticed high values of  significantly perturb semantics (also noted in Garcia et al., 2021), with systems outputting style-specific n -grams unfaithful to the output. We choose  max to be the largest  from the list [0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 , 2 . 5 , 3 . 0] whose outputs have an average semantic similarity score ( SIM , Section 5.3) of at least 0.75 12 with the validation set inputs. For each system we take three evenly spaced  values in its control range, denoted as  = [ 13  max , 23  max ,  max ] . We then compute the style calibration to  ( CALIB ), or how often does increasing  lead to a style score increase? We measure this with a statistic similar to Kendall's  (Kendall, 1938), counting concordant pairs in  , CALIB ( x ) = 1 n (cid:88)  b > a { style ( y  b ) > style ( y  a ) } where x is input, CALIB ( x ) is the average over all possible n ( = 3 ) pairs of  values (  a ,  b ) in  . Automatic metrics are usually insufficient for style transfer evaluation  according to Briakou et al. (2021a), 69 / 97 surveyed style transfer papers used human evaluation. We adopt the crowd-sourcing setup from Section 5.1, which was used to build our formality evaluation datasets. We presented 200 generations from each model and the corresponding inputs in a random order, and asked three crowdworkers two questions about each pair of sentences: (1) which sentence is more formal/code-mixed? (2) how similar are the two sentences in meaning? This lets us evaluate rACC , SIM , rAGG , CALIB with respect to human annotations instead of classifier predictions. More experiment details (inter-annotator agreement, compensation, instructions) are provided in Appendix E.4. Our models are evaluated on (1) formality transfer (Rao and Tetreault, 2018); (2) code-mixing addition, a task where systems attempt to use English words in non-English sentences, while preserving the original script. 13 Since we do not have access to any formality evaluation dataset, 14 we hold out 22K sentences from Samanantar in each Indic language 13 Hinglish is common in India, examples in Figure 5. 14 We do not use GYAFC (Rao and Tetreault, 2018) and XFORMAL (Briakou et al., 2021b) due to reasons in footnote 4. Our dataset from Section 5.1 has already been used for classifier selection, and has machine generated sentences. for validation / testing. For Swahili / Spanish, we use mC4 / WMT2018 sentences. These sets have similar number of formal / informal sentences, as marked by our formality classifiers (Section 5.2), and are transferred to the opposite formality. We re-use the hi/bn formality transfer splits for code-mixing addition, evaluating unidirectional transfer. Seven languages with varying scripts and morphological richness are used for evaluation ( hi,es,sw,bn,kn,te,gu ). The UR model only saw translation data for hi,es,bn , whereas UR-INDIC sees translation data for all Indic languages (Section 4.3). To test the generalization capability of the DIFFUR , no Gujarati paraphrase training data for is used. Note that no paired/unpaired data with style labels is used during training : models determine the target style at inference using 3-10 exemplars sentences. For few-shot formality transfer, we use the English exemplars from Garcia et al. (2021). We follow their setup and use English exemplars to guide non-English transfer zero-shot. For code-mixing addition, we use Hindi/English code-mixed exemplars in Devanagari (shown in Appendix D). automatic evaluation results for formality transfer across languages in Table 1, Table 3. Overall we find that each of our proposed methods ( DIFFUR , *INDIC , + BT ) helps improve performance over the baseline UR model (71.1, 58.3, 54.2 vs 30.4 rAGG on Hindi). Combining these ideas with multitask learning ( DIFFUR-MLT ) gives us the best performance across all languages (78.1 on Hindi). On Gujarati, the DIFFUR-INDIC fails to get good performance (36.0 rAGG ) since it did not see Gujarati paraphrase data, but this performance is recovered using DIFFUR-MLT (75.0). In Table 4 we see human evaluations support our automatic evaluation for formality transfer. In Table 5 we perform human evaluation on a subset of models for code-mixing addition and see similar trends, with DIFFUR-MLT significantly outperforming UR , UR Model Hindi Bengali ACC / SIM / AGG ACC / SIM / AGG UR (2021) 4.5 / 93.8 / 3.6 0.0 / 96.4 / 0.0 UR-INDIC , BT 18.5 / 79.2 / 15.3 18.0 / 68.3 / 12.7 DIFFUR-MLT , BT 62.5 / 69.9 / 41.5 79.0 / 57.1 / 43.5 Table 5: Human evaluation on code-mixing addition. DIFFUR-MLT and DIFFUR-INDIC are best at controlling magnitude of style transfer : In Table 6, we compare the extent to which models can control the amount of style transfer using  . We find that all our proposed methods outperform the UR model, which gets only 29.2 CALIB . + BT models are not as effective at control (43.4 CALIB ), while DIFFUR-INDIC and DIFFUR-MLT perform best (69.6, 69.0 CALIB ). This is graphically illustrated in Figure 3. DIFFUR-MLT performs consistently well across different  values (left plot), and gives a high style change without much drop in content similarity to the input as  is varied (right plot); more control experiments in Appendix F. In Table 2 we provide a breakdown by individual metrics . In the baseline Hindi UR model, we notice high COPY rates (45.4%), resulting in lower 7446 Input Generations Analysis Informal       . ACC scores. COPY reduces in our proposed models (4.4% for DIFFUR-MLT ), which boosts overall performance. We find the lowest COPY (and lowest 1-g) for models with + BT (1%), which is due to two translation steps. However, this lowers semantic similarity (also seen in Table 4) lowering the overall score (60.0 vs 78.1) compared to DIFFUR-MLT . In Appendix G we show ablations studies justifying the DIFFUR design, decoding scheme, etc. In Appendix I we show a breakdown by individual metrics for other languages and plot variations with  . We also analyze the style encoder f style in Appendix H, finding it is an effective style classifier. We analyze several qualitative outputs from DIFFUR-MLT in Figure 4. Besides formality transfer and code-mixing addition, we transfer several other attributes: sentiment (Li et al., 2018), simplicity (Xu et al., 2015), anonymity (Anandan et al., 2012) and gender neutrality (Reddy and Knight, 2016). More outputs are provided in Appendix J. 7 Conclusion We present a recipe for building & evaluating controllable few-shot style transfer systems needing only 3-10 style examples at inference, useful in low-resource settings. Our methods outperform prior work in formality transfer & code-mixing for 7 languages, with promising qualitative results for several other attribute transfer tasks. Future work includes further improving systems for some attributes, and studying style transfer for languages where little / no translation data is available. We are very grateful to the Task Mate team (es-pecially Auric Bonifacio Quintana) for their support and helping us crowdsource data and evaluate models on their platform. We thank John Wieting, Timothy Dozat, Manish Gupta, Rajesh Bhatt, Esha Banerjee, Yixiao Song, Marzena Karpinska, Ar-avindan Raghuveer, Noah Constant, Parker Riley, Andrea Schioppa, Artem Sokolov, Mohit Iyyer and Slav Petrov for several useful discussions during the course of this project. We are also grateful to Rajiv Teja Nagipogu, Shachi Dave, Bhuthesh R, Parth Kothari, Bhanu Teja Gullapalli and Simran Khanuja for helping us annotate model outputs in several Indian languages during pilot experiments. This work was mostly done during Kalpesh Krishna (KK)'s internship at Google Research India, hosted by Bidisha Samanta and Partha Talukdar. KK was partly supported by a Google PhD Fellowship. Recent work has highlighted issues of stylistic bias in text generation systems, specifically machine translation systems (Hovy et al., 2020). We acknowledge these issues, and consider style transfer and style-controlled generation technology as an opportunity to work towards fixing them (for instance, gender neutralization as presented in Section 6.2). Note that it is important to tread down this path carefully  In Chapter 9, Blodgett (2021) argue that style is inseparable from social meaning (as originally noted by Eckert, 2008), and humans may perceive automatically generated text very differently compared to automatic style classifiers. Our models were trained on 32 Google Cloud TPUs. As discussed in Appendix A, the UR & UR-INDIC model take roughly 18 hours to train. The DIFFUR -* and DIFFUR-MLT models are much cheaper to train (2 hours) since we finetune the pretrained UR -* models. The Google 2020 environment report mentions, 15 TPUs are highly efficient chips which have been specifically designed for machine learning applications. These accelerators run on Google Cloud, which is carbon neutral today, and is aiming to run on carbon-free energy, 24/7, at all of Google's data centers by 2030 ( https://cloud.google. com/sustainability ). gumdrop/sustainability/google-2020-environmental-report.pdf
abstain abstain method method objective result method result result method abstain other abstain abstain abstain abstain abstain abstain objective method result objective abstain abstain result method abstain result result method objective other other other other other abstain objective other other objective abstain other other other other other method other other other other other other method abstain other method other abstain other other other other abstain abstain other objective abstain other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain method method abstain abstain abstain method abstain method abstain abstain other abstain method method abstain abstain abstain method abstain method other method method abstain abstain abstain abstain abstain method method method method abstain method method abstain method abstain method abstain method method abstain method abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain result result method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain other other other other other abstain method abstain method abstain method abstain other abstain