The main barrier to progress in the task of Formality Style Transfer is the inadequacy of training data. In this paper, we study how to augment parallel data and propose novel and simple data augmentation methods for this task to obtain useful sentence pairs with easily accessible models and systems. Experiments demonstrate that our augmented parallel data largely helps improve formality style transfer when it is used to pre-train the model, leading to the state-of-the-art results in the GYAFC benchmark dataset 1 . Formality style transfer (FST) is defined as the task of automatically transforming a piece of text in one particular formality style into another (Rao and Tetreault, 2018). For example, given an informal sentence, FST aims to preserve the style-independent content and output a formal sentence. Previous work tends to leverage neural networks (Xu et al., 2019; Niu et al., 2018; Wang et al., 2019) such as seq2seq models to address this challenge due to their powerful capability and large improvement over the traditional rule-based approaches (Rao and Tetreault, 2018). However, the performance of the neural network approaches is still limited by the inadequacy of training data: the public parallel corpus for FST training  GYAFC (Rao and Tetreault, 2018)  contains only approximately 100K sentence pairs, which can hardly satiate the neural models with millions of parameters. To tackle the data sparsity problem for FST, we propose to augment parallel data with three specific data augmentation methods to help improve the model's generalization ability and reduce the over-fitting risk. Besides applying the widely used back  Work done during the internship at Microsoft Research. translation (BT) method (Sennrich et al., 2016a) in Machine Translation (MT) to FST, our data augmentation methods include formality discrimination (F-Dis) and multi-task transfer (M-Task). They are both novel and effective in generating parallel data that introduces additional formality transfer knowledge that cannot be derived from the original training data. Specifically, F-Dis identifies useful pairs from the paraphrased pairs generated by cross-lingual MT; while M-task leverages the training data of Grammatical Error Correction (GEC) task to improve formality, as shown in Figure 1. Experimental results show that our proposed data augmentation methods can harvest large amounts of augmented parallel data for FST. The augmented parallel data proves helpful and significantly helps improve formality style transfer when it is used to pre-train the model, allowing the model to achieve the state-of-the-art results in the GYAFC benchmark dataset. We study three data augmentation methods for formality style transfer: back translation, formality discrimination, and multi-task transfer. We focus on informal  formal style transfer since it is more practical in real application scenarios. The original idea of back translation (BT) (Sen-nrich et al., 2016a) is to train a target-to-source seq2seq (Sutskever et al., 2014; Cho et al., 2014) model and use the model to generate source language sentences from target monolingual sentences, establishing synthetic parallel sentences. We generalize it as our basic data augmentation method and use the original parallel data to train a seq2seq model in the formal-to-informal direction. Then, we can feed formal sentences to this model that is supposed to be capable of generating their informal counterparts. The formal input and the informal output sentences can be paired to establish augmented parallel data. According to the observation that an informal sentence tends to become a formal sentence after a round-trip translation by MT models that are mainly trained with formal text like news, we propose a novel method called formality discrimination to generate formal rewrites of informal source sentences by means of cross-lingual MT models. A typical example is shown in Figure 2. To this end, we collect a number of potentially informal English sentences (e.g., from online fo-rums). Formally, we denote the collected sentences as S = { s i } |S| i =1 where s i represents the i -th sentence. We first translate 2 them into a pivot language (e.g., French) and then translate them back into English, as Figure 2 shows. In this way, we obtain a rewritten sentence s (cid:48) i for each sentence s i  S . To verify whether s (cid:48) i improves the formality compared to s i , we introduce a formality discriminator which in our case is a Convolutional Neural Network (CNN) to quantify the formality level of a sentence. We trained the formality discriminator with the sentences and their formality labels in the FST corpus (e.g., GYAFC). The pairs ( s i , s (cid:48) i ) where s (cid:48) i largely improves the formality of s i will 2 https://translate.google.com/ Input i'm gonna trust my gut feelings. (0.12) Output I will trust my instinct. ( 0.96 ) French je vais faire confiance  mon instinct. MT MT Figure 2: Formality discrimination for FST. be selected as the augmented data. The resulting data set T aug is such a set of pairs: T aug = { ( s i , s (cid:48) i ) | P + ( s (cid:48) i )  P + ( s i )   } (1) where P + ( x ) is the probability of sentence x being formal, predicted by the discriminator, and  is the threshold 3 for augmented data selection. In this way, we can obtain much helpful parallel data with valuable rewriting knowledge that is not covered by the original parallel data. In addition to back translation and formality discrimination that use artificially generated sentence pairs for data augmentation, we introduce multitask transfer that uses annotated sentence pairs from other seq2seq tasks. We observe that informal texts are usually ungrammatical while formal texts are almost grammatically correct. Therefore, a desirable FST model should possess the ability to detect and rewrite ungrammatical texts, which has been verified by the previous empirical study (Ge et al., 2019) showing that using a state-of-the-art grammatical error correction (GEC) model to post-process the outputs of an FST model can improve the result. Inspired by this observation, we propose to transfer the knowledge from GEC to FST by leveraging the GEC training data as the augmented parallel data to help improve formality. An example is illustrated in Figure 1 in which the annotated data for GEC provides knowledge to help the model rewrite the ungrammatical informal sentence. In general, massive augmented parallel data can help a seq2seq model to learn contextualized representations, sentence generation and source-target alignments better. When the augmented parallel 3  = 0 . data is available, previous studies (Sennrich et al., 2016a; Edunov et al., 2018; Karakanta et al., 2018; Wang et al., 2018) for seq2seq tasks are inclined to train a seq2seq model with original training data and augmented data simultaneously. However, augmented data is usually noisier and less valuable than original training data. In simultaneous training, the massive augmented data tends to overwhelm the original data and introduce unnecessary and even erroneous editing knowledge, which is undesirable for our task. To better exploit the augmented data, we propose to first pre-train the model with augmented parallel data and then fine-tune the model with the original training data. In our pre-training & fine-tuning (PT&FT) approach, the augmented data is not treated equally to the original data; instead it only serves as prior knowledge that can be updated and even overwritten during the fine-tuning phase. In this way, the model can better learn from the original data without being overwhelmed or distracted by the augmented data. Moreover, separating the augmented and original data into different training phases makes the model become more tolerant to noise in augmented data, which reduces the quality requirement for the augmented data and enables the model to use noisier augmented data and even training data from other tasks. In this section, we present the experimental settings and related experimental results. We focus on informal  formal style transfer since it is more practical in real application scenarios. We use GYAFC benchmark dataset (Rao and Tetreault, 2018) for training and evaluation. GYAFC's training split contains a total of 110K annotated informal-formal parallel sentences, which are annotated via crowd-sourcing of two domains: Entertainment & Music (E&M) and Family & Relationships (F&R). In its test split, there are 1,146 and 1,332 informal sentences in E&M and F&R domain respectively and each informal sentence has 4 referential formal rewrites. We use all the three data augmentation methods we introduced and obtain a total of 4.9M augmented pairs. Among them, 1.6M are generated by back-translating (BT) formal sentences identified (as formal) by the formality discriminator in E&M and F&R domain on Yahoo Model E&M F&R BLEU BLEU Original data 69.44 74.19 Augmented data 51.83 55.66 ST 59.93 63.16 ST (up-sampling) 68.43 73.04 ST (down-sampling) 68.54 73.69 PT&FT 72.63 77.01 Table 1: The comparison of simultaneous training (ST) and Pre-train & Fine-tuning (PT&FT). Answers L6 corpus 4 , 1.5M are derived by formality discrimination (F-Dis) by using French, German and Chinese as pivot languages, and 1.8M are from multi-task transfer (M-task) from the public GEC data (Lang-8 (Mizumoto et al., 2011; Tajiri et al., 2012) and NUCLE (Dahlmeier et al., 2013)). The informal sentences used in F-Dis strategy are also from Yahoo Answers L6 corpus. We use the Transformer (base) (Vaswani et al., 2017) as the seq2seq model with a shared vocabulary of 20K BPE (Sennrich et al., 2016b) tokens. We adopt the Adam optimizer to pre-train the model with the augmented parallel data and then fine-tune it with the original parallel data. In pre-training, the dropout rate is set to 0.1 and the learning rate is set to 0.0005 with 8000 warmup steps and scheduled to an inverse square root decay after warmup; while during fine-tuning, the learning rate is set to 0.00025. We pre-train the model for 80k steps and fine-tune the model for a total of 15k steps. The CNN we use as the formality discriminator has filter sizes of 3, 4, 5 with 100 feature maps. The dropout rate is set to 0.5. It achieves an accuracy of 93.09% over the GYAFC test set. Table 1 compares the results of the models trained with simultaneous training (ST) and pre-training & fine-tuning (PT&FT). ST with the augmented and original data leads to a performance decline, because the noisy augmented data cannot achieve desirable performance by itself and may distract the model from exploiting the original data in simultaneous training. In contrast, PT&FT only uses 4 https://webscope.sandbox.yahoo.com/catalog.php Model E&M F&R BLEU BLEU Original data 69.44 74.19 Pre-training & Fine-tuning + BT 71.18 75.34 + F-Dis 71.72 76.24 + M-Task 71.91 76.21 + BT + M-Task + F-Dis 72.63 77.01 Table 2: The comparison of different data augmentation methods for FST. the augmented data in the pre-training phase and treats it as the prior knowledge supplementary to the original training data, reducing the negative effects of the augmented data and improving the results. Table 2 compares the results of different data augmentation methods with PT&FT. Pre-training with augmented data generated by BT enhances the generalization ability of the model, thus we observe an improvement over the baseline. However, it does not introduce any new informal-to-formal transfer knowledge, leading to the least improvement among the three methods. In contrast, both F-Dis and M-Task introduce abundant transfer knowledge for FST. The augmented data of F-Dis includes various informal  formal rewrite knowledge derived from the MT models, allowing the model to better handle the test instances whose patterns are never seen in the original training data; while M-Task introduces GEC knowledge that helps improve formality in terms of grammar. We then combine all these beneficial augmented data for pre-training. As expected, the combination strategy achieves further improvement as shown in Table 2 since the it enables the model to take advantage of all the data augmentation methods. We compare our approach to the following previous approaches in the GYAFC benchmark: Rule, PBMT, NMT, PBMT-NMT: Rule-based, phrase-based MT, NMT, PBMT-NMT hybrid model (Rao and Tetreault, 2018). NMT-MTL: NMT model with multi-task learning (Niu et al., 2018). GPT-CAT, GPT-Ensemble: fine-tuned encoder-decoder models (Wang et al., 2019) initialized by GPT (Radford et al., System E&M F&R BLEU BLEU No-edit 50.28 51.67 Rule 60.37 66.40 PBMT 66.88 72.40 NMT 58.27 68.26 NMT-PBMT 67.51 73.78 NMT-MTL 71.29 74.51 NMT-MTL-Ensemble* 72.01 75.33 GPT-CAT 72.70 77.26 GPT-Ensemble* 69.86 76.32 Our Approach 72.63 77.01 Our Approach* 74.24 77.97 Table 3: The comparison of our approach to the state-of-the-art results. * denotes the ensemble results. 2019). Specifically, GPT-CAT concatenates the original input sentence and the input sentence preprocessed by rules as input, while GPT-Ensemble is the ensemble of two GPT-based encoder-decoder models: one takes the original input sentence as input, the other takes the preprocssed sentence as input. Following Niu et al. (2018), we train 4 independent models with different initializations for ensemble decoding. According to Table 3, our single model performs comparably to the state-of-the-art GPT-based encoder-decoder models (more than 200M parameters) with only 54M parameters. Our ensemble model further advances the state-of-the-art result only with a comparable model size to the GPT-based single model (i.e., GPT-CAT). We also conduct human evaluation. Following Rao and Tetreault (2018), we assess the model output on three criteria: formality , fluency and meaning preservation . We compare our baseline model trained with original data, our best performing model and the previous state-of-the-art models (NMT-MTL and GPT-CAT). We randomly sample 300 items and each item includes an input and four outputs that shuffled to anonymize model identities. Two annotators are asked to rate the outputs on a discrete scale of 0 to 2. More details can be found in the appendix. The results are shown in Table 4 which demonstrates that our model is consistently well rated in human evaluation. We also conduct an exploratory study of the pivot languages used in formality discrimination. Among the three pivot languages (i.e. French, German and Chinese) in our experiments, it is interest-Model Formality Fluency Meaning Original data 1.31 1.77 1.80 NMT-MTL 1.34 1.78 1.92 * GPT-CAT 1.42 1.84* 1.90 Ours 1.45 * 1.85 *  1.92 * Table 4: Results of human evaluation of FST. ing to observe a significant difference in the sizes of the obtained parallel data given the same source sentences and filter threshold, as shown in Table 5. Using Chinese as the pivot language results in the most data, probably due to the fact that Chinese and English belong to different language systems. The formality of original informal English sentences may be lost during translation, which turns out to facilitate the MT system to translate Chinese back into formal English. In contrast, French and German have much in common with English, especially for French in terms of the lexicon (Baugh and Cable, 1993). The translated sentences are likely to maintain informal sense, which hinders the MT system from generating formal English translations. We compare the performance with augmented data generated by three pivot languages separately in Table 6. Manual inspection reveals that a few pairs have the issue of meaning inconsistency in all the three sets, which mainly arises from the translation difficulties caused by omissions and poor grammaticality in informal sentences and the segmentation ambiguity in some pivot languages like Chinese. Among the three languages, the Chinese-based augmented data introduces more noise due to the additional segmentation ambiguity problem but brings fair improvement because of its largest size. In contrast, the German-based augmented data has relatively high quality and a moderate size, leading to the best result in our experiments. Data augmentation has been much explored for seq2seq tasks like Machine Translation (He et al., 2016; Fadaee et al., 2017; Zhang et al., 2018b; Pon-Model celas et al., 2018; Edunov et al., 2018; Li et al., 2019) and Grammatical Error Correction (Kiyono et al., 2019; Grundkiewicz et al., 2019; Zhao et al., 2019; Zhou et al., 2019; Ge et al., 2018a,b; Xie et al., 2018; Yuan et al., 2016; Rei et al., 2017). For text style transfer, however, due to the lack of parallel data, many studies focus on unsupervised approaches (Luo et al., 2019; Wu et al., 2019; Zhang et al., 2018a) and there is little related work concerning data augmentation. As a result, most recent work (Jhamtani et al., 2017; Xu et al., 2012) that models text style transfer as MT suffers from a lack of parallel data for training, which seriously limits the performance of powerful models. To solve this pain point, we propose novel data augmentation methods and study the best way to utilize the augmented data, which not only achieves a success in formality style transfer, but also would be inspiring for other text style transfer tasks. In this paper, we propose novel data augmentation methods for formality style transfer. Our proposed data augmentation methods can effectively generate diverse augmented data with various formality style transfer knowledge. The augmented data can significantly help improve the performance when it is used for pre-training the model and leads to the state-of-the-art results in the formality style transfer benchmark dataset. We thank all the reviewers for providing the constructive suggestions. This work is partly supported by Beijing Academy of Artificial Intelligence. Xu Sun is the corresponding author of this paper.
abstain objective objective abstain abstain abstain abstain objective abstain abstain abstain abstain objective abstain method method abstain method method abstain method abstain method method method method method method other abstain abstain abstain abstain abstain abstain other method method method abstain method abstain abstain abstain abstain abstain method method method abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain method abstain method abstain abstain method abstain method method result method method method method abstain abstain method method result abstain abstain abstain abstain abstain method abstain abstain method other other other other objective objective objective abstain other other other