Text style transfer rephrases a text from a source style (e.g., informal) to a target style (e.g., formal) while keeping its original meaning. Despite the success existing works have achieved using a parallel corpus for the two styles, transferring text style has proven significantly more challenging when there is no parallel training corpus. In this paper, we address this challenge by using a reinforcement-learning-based generator-evaluator architecture. Our generator employs an attention-based encoder-decoder to transfer a sentence from the source style to the target style. Our evaluator is an adversarially trained style discriminator with semantic and syntactic constraints that score the generated sentence for style, meaning preservation, and fluency. Experimental results on two different style transfer tasks (sentiment transfer and formality transfer) show that our model outperforms state-of-the-art approaches. Furthermore, we perform a manual evaluation that demonstrates the effectiveness of the proposed method using subjective metrics of generated text quality. Text style transfer is the task of rewriting a piece of text to a particular style while retaining the meaning of the original text. It is a challenging task of natural language generation and is at the heart of many recent NLP applications, such as personalized responses in dialogue system (Zhou et al., 2017), formalized texts (Rao and Tetreault, 2018), cyberspace purification by rewriting offensive texts (Niu and Bansal, 2018; Santos et al., 2018), and poetry generation (Yang et al., 2018). Recent works on supervised style transfer with a parallel corpus have demonstrated considerable success (Jhamtani et al., 2017b; Rao and Tetreault, 2018). However, a parallel corpus may not always be available for a transfer task. This has prompted studies on style transfer without parallel corpora. These hinge on the common idea of separating the content from the style of the text (Shen et al., 2017; Fu et al., 2018; Santos et al., 2018). This line of research first encodes the context via a style-independent representation, and then transfers sentences by combining the encoded content with style information. In addition, an appropriate training loss is chosen to change the style while preserving the content. However, these approaches are limited by their use of loss functions that must be differentiable with respect to the model parameters, since they rely on gradient descent to update the parameters. Furthermore, since focusing only on semantic and style metrics in style transfer, they ignore other important aspects of quality in text generation, such as language fluency. In this paper, we propose a system trained using reinforcement-learning (RL) that performs text style transfer without accessing to a parallel corpus. Our model has a generator-evaluator structure with one generator and one evaluator with multiple modules. The generator takes a sentence in a source style as input and transfers it to the target style. It is an attention-based sequence-to-sequence model, which is widely used in generation tasks such as machine translation (Luong et al., 2015). More advanced model such as graph-to-sequence model can also exploited for this generation task (Xu et al., 2018b). The evaluator consists of a style module, a semantic module and a language model for evaluating the transferred sentences in terms of style, semantic content, and fluency, respectively. Feedback from each evaluator is sent to the generator so it can be updated to improve the transfer quality. Our style module is a style discriminator built using a recurrent neural network, predicting the likelihood that the given input is in the target style. We train the style module adversarially to be a target style classifier while regarding the transferred sentences as adversarial samples. An adversarial training renders style classification more robust and accurate. As for the semantic module, we used word movers' distance (WMD), a state-of-the-art unsupervised algorithm for comparing semantic similarity between two sentences (Kus-ner et al., 2015; Wu et al., 2018b), to evaluate the semantic similarity between input sentences in the source style and the transferred sentences in the target style. We also engaged a language model to evaluate the fluency of the transferred sentences. Unlike prior studies that separated content from style to guarantee content preservation and transfer strength, we impose explicit semantic, style and fluency constraints on our transfer model. Moreover, employing RL allows us to use other evaluation metrics accounting for the quality of the transferred sentences, including non-differentiable ones. We summarize our contributions below: (1) We propose an RL framework for text style transfer. It is versatile to include a diverse set of evaluation metrics as the training objective in our model. (2) Our model does not rely on the availability of a parallel training corpus, thus addressing the important challenge of lacking parallel data in many transfer tasks. (3) The proposed model achieves state-of-the-art performance in terms of content preservation and transfer strength in text style transfer. The rest of our paper is organized as follows: we discuss related works on style transfer in Section 2. The proposed text style transfer model and the reinforcement learning framework is introduced in Section 3. Our system is empirically evaluated on sentiment and formality transfer tasks in Section 4. We report and discuss the results in Section 5 and Section 6. The paper is concluded in Section 7. 2 Related Works Text style transfer has been explored in the context of a variety of natural language applications, including sentiment modification (Zhang et al., 2018b), text simplification (Zhang and Lapata, 2017), and personalized dialogue (Zhou et al., 2017). Depending on whether the parallel corpus is used for training, two broad classes of style transfer methods have been proposed to transfer the text from the source style to the target style. We will introduce each line of research in the following subsections. Style transfer with parallel corpus . Style transfer with the help of a style parallel corpus can be cast as a monolingual machine translation task. For this, a sequence-to-sequence (seq2seq) neural network has been successfully applied in a supervised setting. Jhamtani et al. transfer modern English to Shakespearean English by enriching a seq2seq model with a copy mechanism to replicate the source segments in target sentences (Jhamtani et al., 2017a). Style transfer without parallel corpus . Scarce parallel data in many transfer tasks has prompted a recent interest in studying style transfer without a parallel corpus (e.g., (Zhang et al., 2018a)). Li et al. propose to delete words associated with the source style and replace them with similar phrases associated with the target style. Clearly, this approach is limited to transfers at the lexical level and may not handle structural transfer. Most existing unsupervised approaches share a core idea of disentangling content and style of texts. For a given source sentence, a style-independent content representation is firstly derived. Then, in combination with the target style, the content representation is used to generate the sentence following the target style. Approaches to extract the content include variational auto-encoders (VAE) and cycle consistency. VAEs are commonly used to learn the hidden representation of inputs for dimensionality reduction, and have been found to be useful for representing the content of the source (Hu et al., 2017; Mueller et al., 2017; Shen et al., 2017; Fu et al., 2018). Cycle consistency is an idea borrowed from image style transfer for content preservation (Zhu et al., 2017). It proposes to reconstruct the input sentence from the content representation, by forcing the model to keep the information of the source sentence (Santos et al., 2018). The transferred sentences are generated based on the content representation and the target style. One way to achieve this is with the use of a pre-trained style classifier. The classifier scores the transfer strength of the generated sentences and guides the model to learn the target text style (San-tos et al., 2018; Prabhumoye et al., 2018). Another way is to learn the style embedding, which can be concatenated with the content embedding as the representation of the target sentence (Fu et al., 2018). The decoder then constructs the sentences from their hidden representations. We note that previous works rely on gradient descent in their model training, and therefore their training losses (e.g., content and style loss) were limited to functions differentiable with respect to model parameters. Also, very few works consider other aspects of transfer quality beyond the content and the style of the generated sentences. This is in part due to their reliance on a differentiable training objective. We propose an RL-based style transfer system so that we can incorporate more general evaluation metrics in addition to preserving the semantic meaning of content and style transfer strength. Reinforcement learning . RL has recently been applied to challenging NLP tasks (Yu et al., 2017). RL has advantages over supervised learning in that it supports non-differentiable training objectives and does not need annotated training samples. Benefits of using RL have been demonstrated in image captioning (Guo et al., 2018), sentence simplification (Zhang and Lapata, 2017), machine translation (Wu et al., 2018a) and essay scoring (Wang et al., 2018). A recent work on the task of sentiment transfer applied reinforcement learning to handle its BLEU score-based training loss (a non-differentiable function) (Xu et al., 2018a). Similar to the style transfer works discussed above, it also disentangled the semantics and the sentiment of sentences using a neutralization module and an emotionalization module respectively. Our work is different from these related works in that the semantic preservation and transfer strength are taken care of by the use of discriminators without explicitly separating content and style. An additional aspect that we focus here is the notion of fluency of the transferred sentences, which has not been explored before. Our style transfer system consists of the following modules: a generator, a style discriminator, a semantic module and a language model as shown in Fig. 1. We next describe the structure and function of each component. A closer view of our system is presented in Fig. 2. Generator . transFigure 1: Model overview: the generator transfers the input source sentence to the generated target sentence. The generated sentences are collectively evaluated by the style discriminator, the semantic module and language module respectively. The style discriminator is adversarially trained with both humanand model-generated sentences. These three modules evaluate the generated sentences in terms of transfer strength, content preservation and fluency, and the rewards are sent to train the generator. fers it to the target style. For this, we use a recurrent encoder-decoder model combined with attention mechanism, which can handle variable-length input and output sequences (Sutskever et al., 2014; Cho et al., 2014). We could also leverage recently proposed more advanced encoder-decoder models (Xu et al., 2018b,c) to exploit rich syntactic information for this task, which we leave it as future work. Both the encoder and the decoder are recurrent neural layers with gated recurrent units (GRU). The encoder takes one word from the input at each time step, and outputs a hidden state vector  h s at time s . Similarly, the decoder outputs a hidden representation h t at time t . Suppose that the input sequence consists of T words x = { x 1 , . . . , x T } , and the generated target sentence y is also a sequence of words { y 1 , . . . , y T (cid:48) } . We use vec (  ) to denote the embedding of a word. The gated recurrent unit dynamically updates its state h t based on its previous state h t  1 and current input i t . Its computation can be abstracted as h t = GRU ( h t  1 , i t ) . For the encoder, the input i t is the embedding of the t -th input source word,  h t = GRU ( h t  1 , vec ( x t )) . An attention mechanism is commonly adopted in text generation, such as machine translation (Bahdanau et al., 2015; Luong et al., 2015). We Figure 2: A detailed view of each component in the text style transfer system. apply the attention mechanism to the decoding step so that the decoder learns to attend to source words and generates words. In this work, we use the attention mechanism similar to that used in (Luong et al., 2015). At the t -th decoding step, the attention  t ( s ) is the weight of the s -th encoder state  h ( s ) . The encoder hidden states are linearly weighted by the attention as the context vector at time t . c t = (cid:88)  t ( s ) h s . (3) Combining the attention over the source sentence, the decoder produces a new hidden state  h t ,  h t = tanh ( W c [ c t ; h t ]) . (4) The hidden vector  h t is then used to predict the likelihood of the next word in the target sentence over the target vocabulary. P ( y t | y <t , x ) = softmax ( W s  h t ) . (5) where W c and W s are decoder parameters. Style discriminator . The style discriminator evaluates how well the generated sentences are transferred to the target style. It is a classifier built on a bidirectional recurrent neural network with attention mechanism. The style discriminator is pre-trained to minimize the cross-entropy loss in the style classification task. This style classifier predicts the likelihood that an input sentence is in the target style, and the likelihood is taken as the style score of a sentence. The pre-training does not guarantee that the neural network model will learn robust style patterns. So we resort to adversarial training as done in generative adversarial networks (GAN) (Yu et al., 2017; Wang and Lee, 2018). Accordingly, the style discriminator is later adversarially trained to distinguish the original (human-written) sentences from the model-generated ones so that the classifier learns to classify the text style well. Semantic module . This evaluates how well the content from the input is preserved in the generated sentences. We use word mover's distance (WMD), which is the state-of-the-art approach (known for its robustness and efficiency) to measure the dissimilarity between the input and output sentences based on word embeddings (Kus-ner et al., 2015; Wu et al., 2018b). We take the negative of the WMD distance and divide it by the sequence length to yield the semantic score of a generated sentence. Previous works have also used cycle reconstruction loss to measure content preservation by reconstructing input sentences from generated sentences (Santos et al., 2018). Language model . The style and the semantic modules do not guarantee the fluency of the transferred sentences. This fluency is achieved using a language model. The language model we use is a two-layer recurrent neural network pre-trained on the corpus in the target style so as to maximize the likelihood of the target sentences (Mikolov et al., 2010; Jozefowicz et al., 2016). The language model estimates the probability of input sentences. We take the logarithm of the probability and divide it by the sequence length as the fluency score . The output sentences from the generator are sent to the semantic, style and language model modules for evaluation. These modules give feedback to the generator for the purpose of tuning it and to improve the quality of the generated sentences. We emphasize that despite the fact that our chosen evaluation metrics are not differentiable with respect to the generator parameters, they are still usable here. This is made possible by our use of the RL framework (the REINFORCE algorithm) to update the parameters of the generator (Williams, 1992). In the RL framework, we define the state and the action for our style transfer task as follows. The state s t at time t is the input source sequence and the first t  1 words that are already generated in the target sequence, i.e., s t = ( X, Y 1: t  1 ) . The action a t at time t is the t -th word to be generated in the output sequence, i.e., a t = y t . Suppose that the target vocabulary is V , and the maximum length of the decoder is T (cid:48) . The generator G is parameterized with a parameter set  , and we define the expected reward of the current generator as J ( G  ) . The total expected reward is J ( G  ) = T (cid:48) (cid:88) t =1 EY 1: t  1  G  [ (cid:88) y t  VP  ( y t | s t ) Q ( s t , y t )] , (6) where P  ( y t | s t ) is the likelihood of word y t given the current state, and Q ( s t , y t ) is the cumulative rewards that evaluate the quality of the sentences extended from Y 1: t . Suppose that r ( s t , y t ) is the reward of word y t at state s t . The total reward, Q , is defined as the sum of the word rewards. where  (0 <  < 1) , is a discounting factor so that the future rewards have decreasing weights, since their estimates are less accurate. If we only consider one episode, i.e., Y 1: t  1 has been given for every y t , the reward J ( G  ) can be written as J ( G  ) = T (cid:48) (cid:88) t =1 (cid:88) y t  VP  ( y t | s t ) Q ( s t , y t ) . Sequence sampling . By design, the three evaluation modules in Fig. 1 only evaluate complete sentences instead of single words or partial sentences. This means that we cannot obtain r ( s t , y t ) directly from the evaluation modules at any time instance before the end of the sentence. One way around this problem is rolling out (Yu et al., 2017), where the generator rolls out' the given sub-sentence Y 1: t at time step t to generate complete sentences by sampling the remaining part of the sentence { Y nt +1: T (cid:48) } . Previous works have adopted different sampling strategies, including Monte Carlo search, multinomial sampling and beam search. Starting from the given segment Y 1: t , Monte Carlo search explores the sub-sequence which leads to the best complete sentence (Yu et al., 2017). This leads to a good estimate of the sentence rewards but comes at significant computational cost. In many applications, the other two sampling strategies have been adopted for their efficiency. In multinomial sampling, each word y  ( t <   T (cid:48) ) is sampled from the vocabulary according to the likelihood P ( y  | s  ) predicted by the generator (ODonoghue et al., 2016; Chatterjee and Cancedda, 2010). The beam search process, on the other hand, keeps track of the k (a user-specified parameter) most likely words at each decoding step rather than just one word (Wu et al., 2018a). While this yields an accurate estimate of the reward for each action, multinomial sampling allows us to explore the diversity of generated texts with a potentially higher reward later on. This is the trade-off between exploitation and exploration in RL. To balance the estimation accuracy and the generation diversity, we combine the ideas of beam search and multinomial sampling. Given a source sentence, we first generate a reference target sentence Y ref 1: T using beam search. To estimate the reward at each time step t , we draw samples of complete sentences { Y l 1: T (cid:48) } by rolling out the subsequence Y ref 1: t using multinomial sampling. The evaluation scores of the sampled sentences are used as reward r ( y t , s t ) . More details about the sampling process are in Appendix. Reward estimation . We estimate the reward as follows. We draw N samples of complete sentences starting from Y 1: t : { Y ( n ) 1: T (cid:48) } Nn =1 . The complete sentences are then fed into the three evaluation modules. Let f style be the style score given by the style module, f semantic be semantic score by the semantic module, and f lm be the fluency score given by the language model. We score the action y t at state s t by the average score of the complete sentences rolled out from Y 1: t . This action score is defined as the weighted sum of the scores given by the three modules. f ( s t , y t ) = 1 NN (cid:88) n =1 (   f style ( Y ( n ) 1: T (cid:48) )+   f semantic ( Y ( n ) 1: T (cid:48) , Y real 1: T (cid:48) ) +   f lm ( Y ( n ) 1: T (cid:48) ) ) , (9) where the hyperparameters ,  and  are positive. In our experiments, we set  = 1 . 0 ,  = 0 . 5 and  = 0 . 5 heuristically. We then obtain the discounted cumulative reward Q ( s t , y t ) from the rewards { r ( s  , y  ) } >t at each time step using Eq. 7. The total reward of J ( G  ) can be derived from the cumulative rewards { Q ( s t , y t ) } using Eq. 8. We define the generator loss L  as the negative of reward J ( G  ) , LG (  ) =  J ( G  ) . According to Eq. 8, we can find the gradient   L  of the generator loss as,   LG (  ) =  T (cid:48) (cid:88) t =1   P  ( y t | s t ) Q ( s t , y t ) . The style discriminator is pre-trained on corpora in the source and target styles, and is used to evaluate the strength of style transfer. We note that this pre-training may not be sufficient for the style classifier to learn robust patterns and to provide accurate style evaluation. Indeed, in our experiments we found that even though the generator was trained to generate target sentences by maximizing the style rewards, the one-shot pre-training was insufficient to render the sentences in the target style. Borrowing the idea of adversarial training proposed in GANs, we continuously trained the style discriminator using the generated target sentences. Toward this, we used a combination of a randomly sampled set of human-written target sentences { Y ( k ) human } and model-generated sentences { Y ( k ) model } . Here the model-generated instances act as adversarial training samples, using which, the style discriminator was trained to distinguish the model outputs from human-written sentences. Let the discriminator D be parameterized by a parameter set  . We define the prediction of the style discriminator, D ( Y ) , as the likelihood that the sentence Y is in the target style. The objective of this adversarial training amounts to minimizing the discriminator loss LD : LD (  ) = 1 K (  K (cid:88) k =1 log(1  D  ( Y ( k ) model ))  K (cid:88) k =1 log D  ( Y ( k ) human ) ) . (12) 4 Experiments In this work, we considered two textual style transfer tasks, that of sentiment transfer ( ST , involving negative and positive sentiments) and formality transfer ( FT , involving informal and formal styles) using two curated datasets. We experimented with both transfer directions: positive-to-negative, negative-and-positive, informal-to-formal and formal-to-informal. Dataset . For our experiments with style transfer we used a sentiment corpus and a formality corpus described below. (1) Sentiment corpus. The sentiment corpus consists of restaurant reviews collected from the Yelp website (Shen et al., 2017). The reviews are classified as either negative or positive. (2) Formality corpus. We use the Grammarly's Yahoo Answers Formality Corpus (GYAFC) (Rao and Tetreault, 2018), which is a collection of sentences posted in a question-answer forum (Yahoo Answers) and written in an informal style. In addition, these sentences have been manually rewritten in a formal style. We used the data from the section family and relationships . Note that even though the corpus is parallel, we did not use the parallel information. Table 1 shows the train, dev and test data sizes as well as the vocabulary sizes of the corpora used in this work. Model settings . The word embeddings used in this work were of dimension 50. They were first trained on the English WikiCorpus and then tuned on the training dataset. The width of the beam search (parameter k ) was 8 during the RL and the inference stage. Pre-training . We pre-trained the generator, the style discriminator and the language model before the reinforcement learning stage. We discuss each of these steps below. Generator pre-training . We pre-trained the generator to capture the target style from the respective target corpus. This pre-training occurred before setting up the reward from the evaluator to update its parameters in reinforcement learning. During pre-training, we used a set of target instances with a given instance serving as the input as well as the expected output. Using this set we trained the generator in a supervised manner with the cross-entropy loss as the training objective. Pre-training offered two immediate benefits for the generator: (1) the encoder and decoder learned to capture the semantics and the target style from the target corpus; (2) the generator had a good set of initial parameters that led to faster model training. This second aspect is a significant gain, considering that reinforcement learning is more time consuming than supervised learning. Style discriminator pre-training . The style discriminator in our work was built using a bidirectional recurrent neural network. It was pre-trained using training corpora consisting of sentences in both the source and the target styles. We trained it to classify the style of the input sentences with the cross-entropy classification loss. Language model pre-training . The language model was a two-layer recurrent neural network. Taking a target sentence y = { y 1 , . . . , y T (cid:48) } as the input, the language model predicted the probability of the t -th word y t given the previous subsequence y 1: t  1 . The language model was pre-trained on the training corpus in target style to maximize the probability of y t (1  t  T (cid:48) ) . Baselines . We considered two state-of-the-art methods of unsupervised text style transfer that use non-parallel training corpus. (1) Cross alignment model (CA). The CA model assumes that the text in the source and target style share the same latent content space (Shen et al., 2017). The style-independent content representation generated by its encoder is combined with available style information to transfer the sentences to the target style. We used their publicly available model for ST, and trained the model for FT separately with its default parameters. (2) Multi-decoder seq2seq model (MDS). MDS consists of one encoder and multiple decoders (Fu et al., 2018). Similar to the cross alignment transfer, its encoder learns style-independent representations of the source, and the style specific decoder will rewrite sentences in the target style based on the content representation. We trained the model with its default parameters for both the tasks. We used both automatic and human evaluation to validate our system in terms of content preservation, transfer strength and fluency. Aligning with prior work, we used the automatic metrics of content preservation, transfer and fluency that have been found to be well correlated with human judgments (Fu et al., 2018). For comparison, in Appendix, we also report our style and semantic metrics as provided by the evaluator. Content preservation . A key requirement of the transfer process is that the original meaning be retained. Here we measure this by an embedding based sentence similarity metric s sem proposed by (Fu et al., 2018). The embedding we used was based on the word2vec (CBOW) model (Mikolov et al., 2013). It was first trained on the English WikiCorpus and then tuned on the training dataset. Previous works used pre-trained GloVe embedding (Pennington et al., 2014), but we note that it does not have embeddings for Internet slang commonly seen in sentiment and formality datasets. Transfer strength . The transfer strength s style captures the degree to which the style transfer was Sentiment Negative-to-Positive Positive-to-Negative Metric Content Style Overall Perplexity Content Style Overall Perplexity CA 0.894 0.836 0.432 103.11 0.905 0.836 0.435 185.35 MDS 0.783 0.988 0.437 98.89 0.756 0.860 0.402 156.98 RLS 0.868 0.98 0.460 119.24 0.856 0.992 0.459 174.02 Formality Informal-to-Formal Formal-to-Informal Metric Content Style Overall Perplexity Content Style Overall Perplexity CA 0.865 0.558 0.339 238.05 0.789 0.956 0.432 317.40 MDS 0.519 0.435 0.237 278.65 0.546 0.998 0.353 352.86 RLS 0.885 0.601 0.358 208.33 0.873 0.982 0.462 267.78 Table 3: Automatic evaluation of text style transfer systems on sentiment and formality transfer. carried out and was quantified using a classifier. An LSTM-based classifier was trained for style classification on a training corpus (Fu et al., 2018). The classifier predicts the style of the generated sentences with a threshold of 0.5. The prediction accuracy is defined as the percentage of generated sentences that were classified to be in the target style. The accuracy was used to evaluate transfer strength, and the higher the accuracy is, the better the generated sentences fit in target style. Overall score . We would like to point out that there is a trade-off between content preservation and transfer strength. This is because the outputs resulting from unchanged input sentences show the best content preservation while having poor transfer strength. Likewise, for given inputs, sentences sampled from the target corpora have the strongest transfer strength while barely preserving any content if at all. To combine the evaluation of semantics and style, we use the overall score s overall , which is defined as a function of s sem and s style : s overall = s sem  s style s sem + s style (Fu et al., 2018). Fluency . This is usually evaluated with a language model in many NLP applications (Peris and Casacuberta, 2015; Tuske et al., 2018). We used a two-layer recurrent neural network with gated recurrent units as a language model, and trained it on the target style part of the corpus. The language model gives an estimation of perplexity (PPL) over each generated sentence. Given a word sequence of M words { w 1 , . . . , w M } and the sequence probability p ( w 1 , . . . , w M ) estimated by the language model, the perplexity is defined as: PPL = p ( w 1 , . . . , w M )  1 M . The lower the perplexity on a sentence, the more fluent the sentence is. 4.1.2 Human annotation Noting the best overall score of our system in both directions of the tasks considered (to be discussed in the section that follows), we performed human annotations for content, style and fluency to validate the automatic scores. We chose a sample of 100 sentences generated by our system for each transfer task and collected three human judgments per sentence in each evaluation aspect. The annotation guidelines were: Content preservation . Following the annotation scheme adopted by (Rao and Tetreault, 2018), we asked annotators to rate the semantic similarity between the original and transferred sentence on a scale from 1 to 6. Here 1 means completely dissimilar, 2 means dissimilar but on the same topic, 3 means dissimilar while sharing some content, 4 means roughly similar, 5 means almost similar, and 6 means completely similar. Transfer strength . Annotators were given pairs of original and transferred sentences and were asked to decide which one was more likely to be in the target style. We define transfer strength to be the percentage of transferred sentences that were classified to be in the target style. Fluency . Similar to the annotation of content, annotators scored sentences for fluency on a scale of 1( not fluent ) to 6 ( perfectly fluent ). Some example sentences transferred by our system are shown in Table 2. More transferred sentences generated by our system and those by the baseline methods can be found in the Appendix. We first report the results of the automatic evaluation of our proposed system (denoted as RLS) and the two baselinesthe cross alignment model (CA) (Shen et al., 2017) and the multi-decoder seq2seq model (MDS) (Fu et al., 2018)in Table 3. Sentiment transfer . We notice that CA was the best in preserving content, MDS generated the most fluent target sentences and our model achieved the best trade-off between meaning and Metric Negative-to-positive Positive-to-negative Informal-to-formal Formal-to-informal Content (1-6) 5.19 5.20 4.96 5.33 Style accuracy 0.90 0.91 0.83 0.86 Fluency (1-6) 5.51 5.61 5.33 5.21 Table 4: Human judgments of transferred sentences style with the highest overall score. Looking at the Overall score, it is notable that despite the differences in performance between the models studied here, each one performs similarly in both directions. This could be interpreted to mean that with respect to difficulty of transfer, style transfer is equivalent in both the directions for this task. Formality transfer . For this task, we notice that our model outperforms the baselines in terms of content preservation, transfer strength and fluency with the best Overall score and perplexity. This suggests that our model is better at capturing formality characteristics compared to the baselines. We also note that the style strength of all models for informal-to-formal transfer is significantly lower than that for formal-to-informal transfer. This suggests that the informal-to-formal transfer is harder than the reverse. A plausible explanation is that informal sentences are more diverse and thus easier to generate than formal sentences. For example, informality can be achieved by multiple ways, such as by using an abbreviation (e.g., u used as you) and adding speech markers (e.g., hey and ummm), while formality is achieved in a more restricted manner. Another challenge for informal-to-formal transfer is that informal data collected from online users usually contain non-negligible spelling errors such as defenetely, htink and realy. Words being the smallest semantic units in all the models considered here, these spelling errors could affect the transfer performance. For each direction of transfer, we average the scores by annotators for each evaluation item, and report the results in Table 4. Our transferred sentences are shown to have good quality in content, style and fluency in subjective evaluations. To gain insights into the ways in which our approach performs the intended style transfer, we randomly sampled the generated sentences in the informal-to-formal transfer task. We found that the forms of rewriting can be broadly classified as: lexical substitution, word removal, word insertion and structural change. We show the following examples to these forms of re-writing, where the changed parts are highlighted. (1) Lexical substitution. The informal sentence I do n't know what u mean was transferred to I do not know what you mean; (2) Word removal. The informal sentence  And I dont know what I should do was rewritten as I do not know what I should do; (3) Word insertion. In the example instance de-pends on the woman that was changed to  It depends on the woman, we see that a subject was added to generate a complete formal sentence. (4) Structural change. A small number of instances were also rewritten by making structural changes. For example, the informal sentence  Just tell them , what are they gonna do , slap you ?? was transferred to a formal version as  You should tell them , they can not slap you. Other ways of style transfer by incorporating evaluation metrics of structural diversity are left for future work. We proposed a reinforcement-learning-based text style transfer system that can incorporate any evaluation metric to enforce semantic, stylistic and fluency constraints on transferred sentences. We demonstrated its efficacy via automatic and human evaluations using curated datasets on two different style transfer tasks. We will explore and incorporate other metrics to improve other aspects of generated texts such as the structural diversity in the future work. This work is supported by IBM-ILLINOIS Center for Cognitive Computing Systems Research (C3SR) a research collaboration as part of the IBM AI Horizons Network. We thank the NAACL anonymous reviewers for their constructive suggestions.
abstain abstain method method method result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain method method abstain method method abstain method objective objective abstain objective method abstain method result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective abstain abstain abstain abstain abstain abstain method objective abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain abstain method method abstain abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain method abstain method method abstain method abstain abstain abstain abstain method abstain method method abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain method abstain abstain abstain method abstain abstain result result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result result method result result abstain abstain abstain result abstain abstain abstain abstain objective objective objective other other