Formality style transfer (FST) is a task that involves paraphrasing an informal sentence into a formal one without altering its meaning. To address the data-scarcity problem of existing parallel datasets, previous studies tend to adopt a cycle-reconstruction scheme to utilize additional unlabeled data, where the FST model mainly benefits from target-side unlabeled sentences. In this work, we propose a simple yet effective semi-supervised framework to better utilize source-side unlabeled sentences based on consistency training. Specifically, our approach augments pseudo-parallel data obtained from a source-side informal sentence by enforcing the model to generate similar outputs for its perturbed version. Moreover, we empirically examined the effects of various data perturbation methods and propose effective data filtering strategies to improve our framework. Experimental results on the GYAFC benchmark demonstrate that our approach can achieve state-of-the-art results, even with less than 40% of the parallel data 1 . Formality style transfer (FST) (Rao and Tetreault, 2018) has garnered growing attention in the text style transfer community, which aims to transform an informal -style sentence into a formal one while preserving its meaning. The large amount of user-generated data from online resources like tweets often contain informal expressions such as slang words (e.g., gonna ), wrong capitalization or punc-tuations, and grammatical or spelling errors. FST can clean and formalize such noisy data, to benefit downstream NLP applications such as sentiment classification (Yao and Yu, 2021a). Some examples of FST data are presented in Table 1. With the release of the FST benchmark Gram-marly Yahoo Answers Corpus (GYAFC) (Rao and 1 Code available at https://github.com/Aolius/ semi-fst . Informal TITANIC I THINK IT COST ABOUT 300 MILLION Formal I think that Titanic cost around 300 million dollars. Informal being condiderate of her feelings and needs Formal I am being considerate of her personal needs and feelings. Table 1: Examples of informal-formal sentence pairs. Tetreault, 2018), previous studies on FST tend to employ neural networks such as sequence-to-sequence (seq2seq) models to utilize parallel (infor-mal and formal) sentence pairs. However, GYAFC only contains 100k parallel examples, which limits the performance of neural network models. Several approaches have been developed to address the data-scarcity problem by utilizing unlabeled sentences. In a previous study, Zhang et al. (2020) proposed several effective data augmentations methods, such as back-translation, to augment parallel data. Another line of research (Shang et al., 2019; Xu et al., 2019; Chawla and Yang, 2020) conducted semi-supervised learning (SSL) in a cycle-reconstruction manner, where both forward and backward transfer models were jointly trained while benefiting each other by generating pseudo-parallel data from unlabeled sentences. Under this setting, both additional informal and formal sentences are utilized; however, the forward informal  formal model mostly benefits from the target -side (formal) sentences, which are back-translated by the formal  informal model to construct pseudo training pairs. Conversely, the formal  informal model can only acquire extra supervision signals from informal sentences. Because the main objective of FST is the informal  formal transfer, the additional informal sentences were not well utilized in previous studies. In addition, these semi-supervised models incorporate many auxiliary modules such as style discriminators, to achieve state-of-the-art results, which result in rather complicated frameworks and more model parameters. As noisy informal sentences are easier to ac-4689 quire from online resources, we attempt to take a different view from existing approaches, by adopting additional source -side (informal) sentences via SSL. We gain insights from the state-of-the-art approaches for semi-supervised image and text classification (Sohn et al., 2020; Xie et al., 2020; Berthelot et al., 2019; Zhang et al., 2021; Chen et al., 2020) and propose a simple yet effective SSL framework for FST using purely informal sentences. Our approach employs consistency training to generate pseudo-parallel data from additional informal sentences. Specifically, we enforce the model to generate similar target sentences for an unlabeled source-side sentence and its perturbed version, making the model more robust against the noise in the unlabeled data. In addition, a supervised loss is trained simultaneously to transfer knowledge from the clean parallel data to the unsupervised consistency training. Data perturbation is the key component of consistency training and significantly affects its performance. To obtain a successful SSL framework for FST, we first empirically study the effects of various data perturbation approaches. Specifically, we explore easy data augmentation methods, such as random word deletion , and advanced data augmentation methods, such as back-translation . We also handcraft a line of rule-based data perturbation methods to simulate the features of informal sentences, such as spelling error injection . Furthermore, we propose three data filtering approaches in connection with the three evaluation metrics of FST: style strength, content preservation, and fluency. Specifically, we adopt style accuracy , source BLEU, and perplexity as three metrics to filter out low-quality pseudo-parallel data based on a threshold. We also propose a dynamic threshold algorithm to automatically select and update the thresholds of source-BLEU and perplexity. We evaluate our framework on the two domains of the GYAFC benchmark: Entertainment & Music (E&M) and Family & Relationships (F&R). We further collect 200k unpaired informal sentences for each domain to perform semi-supervised training. Experimental results verify that our SSL framework can enhance the performance of the strong supervised baseline, a pretrained T5-large (Raf-fel et al., 2020) model, by a substantial margin, and improve the state-of-the-art results by over 2.0 BLEU scores on both GYAFC domains. Empirically, we also deduce that simple word-level data augmentation approaches are better than advanced data augmentation methods that excessively alter the sentences, and spelling error injection is especially effective. In addition, our evaluation-based data filtering approach can further improve the performance of the SSL framework. Furthermore, we also conduct low-resource experiments by reducing the size of parallel data. Surprisingly, our framework could achieve the state-of-the-art results with only less than 40% of parallel data, demonstrating the advantage of our method in low-resource situations. Formality style transfer FST is an important branch of text style transfer. For FST, Rao and Tetreault (2018) released a high-quality parallel dataset GYAFC, comprising two sub-domains and approximately 50k parallel data for each domain. Previous studies (Rao and Tetreault, 2018; Niu et al., 2018; Xu et al., 2019; Zhang et al., 2020) typically train seq2seq encoder-decoder models on this benchmark. Recent studies (Wang et al., 2019; Yao and Yu, 2021b; Chawla and Yang, 2020; Lai et al., 2021) have deduced that fine-tuning large-scale pretrained models such as GPT-2 (Radford et al., 2019) and BART (Lewis et al., 2020) on the parallel corpora can improve the performance. To address the data-scarcity problem of parallel datasets, Zhang et al. (2020) proposed three data augmentation techniques to augment pseudo-parallel data for training. Similar to prior research on text style transfer that adopt back-translation (Zhang et al., 2018; Lample et al., 2018; Prabhumoye et al., 2018; Luo et al., 2019), some other approaches on FST (Shang et al., 2019; Xu et al., 2019; Chawla and Yang, 2020) adopt a cycle-reconstruction scheme, where an additional backward transfer model is jointly trained together with the forward transfer model, and the two models generate pseudo-paired data for each other via iterative back-translation. Although Xu et al. (2019) and Chawla and Yang (2020) train a single model to perform bidirectional transfer, the generation of both directions remain disentangled by a control variable, making each direction rely on the unlabeled data of its target side. Therefore, the unlabeled informal sentences exert no direct effects on the informal  formal transfer. In contrast, our work focuses on how to better utilize source-side unlabeled data (i.e., informal sentences) using SSL and does not introduce any extra models. SSL with consistency regularization SSL is popular for its advantage in utilizing unlabeled data. Consistency regularization (also known as consistency training) (Sajjadi et al., 2016) is an important component of recent SSL algorithms on image and text classification (Miyato et al., 2018; Tarvainen and Valpola, 2017; Berthelot et al., 2019; Sohn et al., 2020). It enforces a model to produce invariant predictions for an unlabeled data and its perturbed version. These studies developed different data perturbation (Xie et al., 2020; Berthelot et al., 2019) or data filtering (Zhang et al., 2021; Xu et al., 2021) approaches to improve the performance. However, few studies have been made on how to apply consistency training in natural language generation (NLG) tasks such as FST because of the different target spaces, i.e., instead of single class labels or probabilities, the output of NLG is the combination of discrete NL tokens. This renders the experiences in classification tasks not applicable to FST. For instance, classification probabilities are typically adopted as the metric to filter high-confidence pseudo-examples for consistency training in classification tasks (Sohn et al., 2020; Xie et al., 2020; Zhang et al., 2021), which is implausible in FST. A similar study (He et al., 2019) improved self-training by injecting noise into unlabeled inputs and proved its effectiveness on machine translation and text summarization; however, self-training involves multiple iterations to collect pseudo-parallel data and retrain the model, hence the training is not end-to-end. In this study, we explore various data perturbation strategies and propose effective data filtering approaches to realize a successful consistency-based framework for FST, which may also provide useful insights for future studies on semi-supervised NLG. FST involves rewriting an informal sentence into a formal one. Formally, given a sentence x = ( x 1 , x 2 , . . . , x n ) of length n with style S , our objective is to transform it into a target sentence y = ( y 1 , y 2 , . . . , y m ) of length m and style T , while preserving its content. Following prior studies (Rao and Tetreault, 2018; Zhang et al., 2020; Chawla and Yang, 2020; Lai et al., 2021) on FST, we employ the supervised baseline as a seq2seq encoder-decoder model that directly learns the conditional probability P ( y | x ) from parallel corpus D comprising ( x , y ) pairs. The objective is the cross-entropy loss between the decoder outputs and the ground-truth target sentences: L sup = E ( x , y )  D [  log P ( y | x ;  )] = E ( x , y )  D [  (cid:88) i log P ( y i | y 1: i  1 , x ;  )] , (1) where  denotes the model parameters. Our approach leverages the idea of consistency regularization (Sajjadi et al., 2016) and enforces model to generate similar target sentences for an original and perturbed unlabeled sentence. Simultaneously, the model is also trained on the supervised data. Accordingly, the knowledge garnered from supervised training can be gradually transferred to unsupervised training. An overview of our framework is presented in Figure 1. Typically, the consistency training loss is computed on the divergence between predictions on an unlabeled input u and its perturbed version  u = c ( u ) , where c (  ) is the perturbation function and u  US represents a source-side unlabeled sentence (in our case, an informal sentence). Formally, consistency training can be defined as minimizing the following unsupervised loss: E u U SD [ P ( y | u ;  ) || P ( y | c ( u );  )] , (2) where D [ || ] denotes a divergence loss. In practice, we adopt pseudo-labeling (Lee et al., 2013) to train the unsupervised loss, for which we fix the model parameter  to predict a hard label (pseudo target sentence)  y for u and enforce the consistency of model prediction by training  with ( c ( u ) ,  y ) . Hence the unsupervised objective can be optimized as a standard cross-entropy loss as follows: L unsup = E u U SE  y  P ( y | u ;  ) [  log P ( y | c ( u );  )] , (3) where   denotes a fixed copy of  . This training process does not introduce additional model parameters. The entire additional training cost to supervised learning is a training pass and a generation pass for each unlabeled sentence. As the overall objective, we train a weighted sum of the supervised loss in Equation (1) and the unsupervised loss in Equation (3): L = L sup +  L unsup , (4) where  represents a hyper-parameter for balancing the effects of supervised and unsupervised training. To achieve a good initial model for consistency training, we first pretrain the model on the supervised loss for several warm-up steps. Data perturbation is the key component of consistency-based SSL algorithms (Xie et al., 2020; Chen et al., 2020) and significantly affects the performance. In this section, we briefly introduce a collection of different data perturbation methods explored in this research. First, we consider some easy data augmentation methods commonly used for supervised data augmentation, which includes  word deletion (drop) 2 : to randomly drop a proportion of words in the sentence. word swapping (swap) : to randomly swap a proportion of words with their neighbouring words. word masking (mask) : to randomly replace words with a mask token _. word replacing with synonym (synonym) : to randomly replace some words with a synonym based on WordNet (Fellbaum, 1998). back-translation : to translate a sentence into a pivot language, then translate it back to obtain a paraphrase of the original one. TF-IDF based word replacing (tf-idf) : to replace uninformative words with low TF-IDF scores while retaining those with high TF-IDF values. Furthermore, we handcraft a set of rule-based data perturbation for FST. There are some typical informal expressions in the parallel corpus, such as the use of slang words and abbreviations, capitalized words for emphasis, and spelling errors. Some existing studies (Wang et al., 2019; Yao and Yu, 2021b) adopt editing rules to revise such informal expressions as a preprocessing step. Inspired by these, we propose the adoption of opposite rules to synthesize such noises. We consider the following methods:  spelling error injection (spell) : to randomly inject spelling errors to a proportion of words by referring to a spelling error dictionary. word replacing with abbreviations (abbr) : to replace all the words in the sentence with their abbreviations or slang words (e.g., are you  r u) by referring to an abbreviation dictionary. capitalize a proportion of words. 2 We abbreviate each method for ease of denotation. These rule-based methods can inject noise into the unlabeled informal sentences without changing its informality, but strengthening it instead. In the consistency training loss, the noisy pseudo-target  y is generated from the decoder model and may exert negative effects on the training. Therefore, we propose three evaluation-based data filters in connection with the evaluation metrics of FST. Specifically, we attempt to measure the quality of pseudo-target sentences by considering the three most important evaluation criteria of text style transfer: style strength , content preservation , and fluency . Next, we comprehensively explain each evaluation metric and the corresponding data filter. Style strength measures the formality of generated sentences. Typically, people adopt binary classifiers such as TextCNN (Chen, 2015) classifiers to judge the formality of a sentence (Lai et al., 2021). Inspired by this, we pretrain a TextCNN formality classifier on the parallel training corpus (i.e., GYAFC) to distinguish between informal and formal sentences. For an unlabeled informal sentence u and its pseudo target sentence  y , we maintain ( c ( u ) ,  y ) for unsupervised training only when p + cls ( y )  p + cls ( u ) > , (5) where p + cls (  ) represents the probability of the sentence being formal, predicted by the style classifier and  is a threshold of the probability. This guarantees that only the sentence pairs with strong style-differences are used for consistency training. Content preservation is another important evaluation metric of FST, typically measured with BLEU between the ground-truth target sentence and the model generations. In unsupervised text style transfer where no ground-truth target exists, source -BLEU is adopted as an alternative, i.e., the BLEU scores between the source input sentence and the generated target sentence. Similarly, we propose the adoption of source -BLEU between u and  y as the metric to filter out pseudo targets that present poor content preservation. Fluency is also used to evaluate the quality of generated sentences. We follow (Hu et al., 2020) to pretrain an N-gram language model on the training data to estimate the empirical distributions of formal sentences. Then, the perplexity score is calculated for the pseudo target sentence  y by the language model. The motivation is that the sentences with lower perplexity scores match the empirical distribution of formal sentences better, and are thus considered as more fluent. A natural idea is to filter out pseudo-parallel data based on a source -BLEU or a perplexity threshold. However, it is infeasible to determine the optimal threshold for the two metrics beforehand because the pseudo paired data are generated on-the-fly during the training and we cannot know the distribution of the BLEU or perplexity scores. In addition, choosing the BLEU/perplexity threshold is not as easy as tuning the style probability  because they heavily depend on the data distribution and exhibit varying ranges of values. To realize the selection of thresholds for the BLEU-and perplexitybased filters, we propose a dynamic threshold strategy based on the distribution of the scores computed for already generated pseudo-paired sentences. Specifically, we maintain an ordered list L to store the scores calculated for previously generated pseudo data and update it continuously following the training. At each iteration, a batch of new scores are inserted into L while maintaining the decreasing order of the list. Subsequently, we update the threshold as the value at a certain position L [   len ( L )] in the score list, where len ( L ) denotes the length of the current score list and   [0 , 1] represents a ratio that determines the threshold's position in the list. We only keep pseudo data with scores higher (or lower for perplexity scores) than the threshold for consistency training. This actually makes  approximately the proportion of pseudo data we keep for training, making it more convenient to control the trade-off between the qualities and quantities of selected pseudo data. More details are provided in Appendix B, C. 4 Experiments We introduce the experimental settings in Section 4.1. To obtain relevant findings on how to build an effective consistency training framework for FST, we first empirically study the effects of multiple data perturbation methods in Section 4.2 and prove the effectiveness of consistency training via comparisons with the base model. Then, we validate our consistency training model with different data filtering methods in Section 4.3 and demonstrate their additional effects on the SSL frame-4693 Dataset Train Val Test Unlabeled E&M 52595 2877 1416 200k F&R 51967 2788 1432 200k Table 2: The statistics of datasets. work. Based on the findings in these two experiments, we further compare our best models with previous state-of-the-art models in Section 4.4. We also include case studies in Section 4.4 to present some qualitative examples. Finally, we conduct low-resource experiments (Section 4.5) to demonstrate our method's advantage when less parallel data are available. Datasets We evaluate our framework on the GYAFC (Rao and Tetreault, 2018) benchmark for formality style transfer. It comprises crowdsourced informal-formal sentence pairs split into two domains, namely, E&M and F&R. The informal sentences in the dataset were originally selected from the same domains in Yahoo Answers L6 corpus 3 . We focus on the informal-formal style transfer because it is more realistic in applications. We further collected massive amounts of informal sentences from each of the two domains in Yahoo Answers L6 corpus as the unsupervised data. The statistics of the datasets are presented in Table 2. Implementation Details We employ PyTorch (Paszke et al., 2019) for all the experiments. We pretrain a TextCNN style classifier on the supervised data for each domain of GYAFC, following the setting in (Lai et al., 2021). The same classifier is adopted for both the style accuracy evaluation and the style strength filter in our SSL framework. We adopt HuggingFace Transformers (Wolf et al., 2020) library's implementation of pretrained T5-Large (Raffel et al., 2020) as the base model. We adopt the Adam (Kingma and Ba, 2014) optimizer with the initial learning rate 2  10  5 to train all the models. More details of hyper-parameters and model configurations are provided in Appendix A. Evaluation Metrics The main evaluation metric for FST is the BLEU score between the generated sentence and four human references in the test set. We adopt the corpus BLEU in NLTK (Loper and Bird, 2002) following (Chawla and Yang, 2020). In addition, we also pretrained a TextCNN formality 3 https://webscope.sandbox.yahoo.com/catalog.php classifier to predict the formality of transferred sentences and calculate the accuracy (Acc.). Furthermore, we compute the harmonic mean of BLEU and style accuracy as an overall score, following the settings in (Lai et al., 2021). In this experiment, we validate the effectiveness of our consistency training framework and compare the effects of different data perturbation methods. Specifically, we adopt the nine data perturbation methods introduced in Section 3.3 and include the no-perturbation variant that indicates directly using an unlabeled sentence and its pseudo target to train the unsupervised loss. We adopted no data filtering strategy in this experiment to simplify the comparison. As shown in Table 3, our framework could consistently improve the base model by using different perturbation methods; however, back-translation resulted in mostly lower results than the base model. This contradicts the conclusion in (Xie et al., 2020) that back-translation is especially powerful for semi-supervised text classification. We attribute this to the fact that back-translation tends to change the entire sentence into a semantically similar but syntactically different sentence. Compared with other word-level perturbation strategies, back-translation triggers a larger mismatch between the perturbed input and the pseudo-target sentence generated from the unperturbed input, leading to a poor content preservation ability of the model. In contrast, simple word-level noises achieved consistently better results, especially spell error ( spell ), random word swapping ( swap ), and abbreviation replacing ( abbr ). These three methods tend to alter the words but do not lose their information while other methods eliminate the entire word by deleting ( drop, mask ) or replacing it with another word ( synonym, tf-idf ). This may also cause a larger mismatch between the pseudo input and output. Hence, we draw the conclusion that simple word-level perturbations tend to bring more effects . This differs from the observations in text classification (Xie et al., 2020) because content preservation is important in FST. In particular, we also found that spell achieved the highest BLEU scores on both datasets. However, adding no perturbation even resulted in a worse performance than the base model. Moreover, capital is also relatively weaker than the other two rule-based methods because it 4694 E&M F&R Model variants BLEU Acc(%) HM BLEU Acc(%) HM base model 76.87 90.04 82.94 80.32 84.01 82.12 no-perturbation 76.41 88.49 82.01 79.22 84.46 81.75 drop 77.55 93.15 84.64 80.53 86.56 83.44 swap 77.90 93.43 84.96 81.07 85.96 83.44 mask 77.52 93.93 84.94 80.69 86.41 83.45 synonym 77.48 93.64 84.80 80.49 86.26 83.28 back-translation 76.07 90.11 82.50 79.96 84.91 82.36 tf-idf 76.89 92.58 84.01 80.48 86.94 83.58 abbr 77.55 93.64 84.84 81.00 86.94 83.86 capital 77.54 93.15 84.63 80.74 85.74 83.16 spell 78.37 94.21 85.56 81.09 85.59 83.28 Table 3: Effects of different data perturbations in our approach on the test splits of GYAFC. only changes the case of a chosen word. This suggests that the perturbation should not be too simple either. In this section, we analyze whether our proposed data filters are beneficial to the performance of our consistency training framework. Specifically, we chose the most effective data perturbation method spell to analyze the effects of adding the three data filters: style strength ( style ), content preservation ( bleu ), and fluency ( lm ) filters. As presented in Table 4, the results for different datasets and different filters have different tendencies. For example, adding the style filter on the E&M dataset caused negative effects while contributing the best results to the F&R domain. Although a filter does not necessarily improve the result, this is reasonable because filters result in less pseudo data for model training and it is difficult to control the trade-off between the quality and the quantity of selected data. Nevertheless, we still observe that the bleu filter contributes to the highest performance of spell for all the metrics on the E&M domain, while style benefits the performance of spell the most on F&R, leading to the best performing models of our approach 4 . We compare our best model with the following previous studies on GYAFC. NMT (Rao and Tetreault, 2018) is an LSTM-based encoder-decoder model with attention. GPT-CAT (Wang et al., 2019) adopts GPT-2 and rule-based pre-processing for informal sentences. NMT-Multi-task (Niu et al., 2018) jointly solves monolingual formality transfer and formality-sensitive machine translation via multi-task learning. Hybrid Annotations (Xu et al., 2019) trains a CNN discriminator in addition to the transfer model and adopts a cycle-reconstruction loss to utilize unsupervised data. Transformers (DA) (Zhang et al., 2020) uses three data augmentation methods, includ-4 Empirically, we also found that mixing up three filters achieved no better results than a single filter, possibly because this filtered out too much pseudo data. ing back-translation, formality discrimination, and multi-task transfer. CARI (Yao and Yu, 2021b) improves GPT-CAT by using BERT (Devlin et al., 2018) to select optimal rules to pre-process the informal sentences. Chawla's (Chawla and Yang, 2020) uses language model discriminators and maximizing mutual information to improve a pretrained BART-Large (Lewis et al., 2020) model, along with a cycle-reconstruction loss to utilize unlabeled data. BART-large+SC+BLEU (Lai et al., 2021) improves BART-large by incorporating reinforcement learning rewards to enhance style change and content preservation. We also report the results of Ours (base) , our backbone T5-large model, and Ours (best) , our best performing models selected from Table 4. As observed in Table 5, Ours (best) outperforms previous state-of-the-art models by a substantial margin and improves the BLEU scores from 76.17 and 79.92 to 78.75 and 81.37, respectively, on the E&M and F&R domains of the GYAFC benchmark. Although BART-large+SC+BLEU achieved better results on the Acc. of F&R, the only released official outputs of BART-large+SC+BLEU were obtained from a model that was trained on the training data of both domains and adopted rewards to directly optimize style accuracy; hence, it is not directly comparable to our model. Ours (best) improves the fine-tuned T5-large baseline by a large margin as well, demonstrating the effectiveness of our SSL framework. Human Evaluation We also conduct human evaluation to better capture the quality of the mod-els' outputs. Following (Zhang et al., 2020), we measure the Formality , Fluency , and Meaning Preservation of generated sentences by asking two human annotators to assign a score ranging from {0, +1, +2} regarding each aspect. We randomly sampled 50 examples from the test set of each domain and compare the generated outputs of Ours (base) , Ours (best) , and the previous state-of-the-art Chawla's model trained on the single-domain data. In addition, the annotators were unaware of the corresponding model of each output. As shown 4696 Example 1 Source I like natural / real girls, I don't like fake looking prissy drama queens. in Table 6, the human evaluation results are consistent with the automatic evaluation results: Ours (base) is competitive compared with Chawla's , while Ours (best) improves over the base model and outperforms the previous state-of-the-art on all the metrics, except that it presents lower results on Meaning than Ours (base) on F&R. More details on human evaluation can be found in Appendix D. Qualitative Examples We present some of the generated outputs of Ours (base) , Ours (best) , and Chawla's in Table 8. It can be observed that all the models can produce high-quality outputs with considerable formality, meaning preservation and fluency. Nevertheless, Ours (best) exhibits a stronger capability to modify the original sentence, especially for some informal expressions, leading to the best performance on the Formality metric. For example, it replaced like with similar to in Example 2 and deleted the informal word guys in Example 3. However, it may alter the original sentence so much that the meaning of the sentence is changed to some extent (Example 1). This may explain why Ours (best) achieves a lower Meaning score than Ours (base) on F&R. We also simulate the low-resource settings by further reducing the size of available parallel data. Specifically, we randomly sample from the original training data with a size in the range of {100, 1000, 5000, 20000} and compare the results of the base model T5-Large with our SSL model. The size of unlabeled data remains 200k for each domain. We adopt the spell data perturbation without any data filter and avoid exhaustive hyper-parameter tuning. Table 7 demonstrates that our framework is especially effective under few-shot settings when only 100 parallel data are available. By comparing with previous state-of-the-art results on FST, we can observe that our approach can achieve competitive results with only 5000 ( < 10% ) parallel training data, and even better results with only 20000 ( < 40% ) parallel examples. In this study, we proposed a simple yet effective consistency-based semi-supervised learning framework for formality style transfer. Unlike previous studies that adopted cycle-reconstruction to utilize additional target-side sentences for back-translation, our method offers a different view, to leverage source-side unlabeled sentences. Without introducing additional model parameters, our method can easily outperform the strong supervised baseline and achieve the new state-of-the-art results on formality style transfer datasets. For future work, we will attempt to generalize our approach to other text generation scenarios. This paper is based on results obtained from a project, JPNP18002, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). Ao Liu acknowledges financial support from the Advanced Human Resource Development Fellowship for Doctoral Students, Tokyo Institute of Technology.
abstain abstain objective objective objective objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain objective objective method objective method objective method method result abstain result method objective other other other other other other other other objective other other other other other other other other objective abstain method method other method abstain abstain abstain method method abstain method abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method method method abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain method abstain method method abstain method method method method method method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method method method method abstain method abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result objective method objective abstain other other