In this paper, we introduce a new task, Contextual Text Style Transfer, to translate a sentence within a paragraph context into the desired style (e.g., informal to formal, offensive to non-offensive). Two new datasets, Enron-Context and RedditContext, are introduced for this new task, focusing on formality and offensiveness, respectively. Two key challenges exist in contextual text style transfer: 1) how to preserve the semantic meaning of the target sentence and its consistency with the surrounding context when generating an alternative sentence with a specific style; 2) how to deal with the lack of labeled parallel data. To address these challenges, we propose a Context-Aware Style Transfer (CAST) model, which leverages both parallel and non-parallel data for joint model training. For parallel training data, CAST uses two separate encoders to encode each input sentence and its surrounding context, respectively. The encoded feature vector, together with the target style information, are then used to generate the target sentence. A classifier is further used to ensure contextual consistency of the generated sentence. In order to leverage massive non-parallel corpus and to enhance sentence encoder and decoder training, additional self-reconstruction and back-translation losses are introduced. Experimental results on Enron-Context and Reddit-Context demonstrate the effectiveness of the proposed model over state-of-the-art style transfer methods, across style accuracy, content preservation, and contextual consistency metrics.
The authors propose the task of contextual text style transfer: transferring the style of one text into another (i.e., informal to formal, or offensive to non-offensive), when the text is present within some larger, provided context. The authors propose a model (CAST) which takes advantage of the additional context to perform the style transfer. CAST outperforms previous style transfer models according to several automatic metrics, as well as human evaluation.
Overall, I am not convinced that the context is useful in the proposed tasks (Reddit-context and Enron-context), for 2 reasons:
* In the examples shown in Table 4, it seems possible to complete the proposed contextual style transfer tasks without the additional context (i.e., "Do y’all interface with C/P" -> "Do you all interface with C/P" for formal->informal transfer).
* CAST performs better across several metrics (including human evaluation), but I am not sure that the gain is from using the context in the manner that the authors suggest (increasing the generated sentence's coherence with the context). In particular, it is hard for me to see how some style transferred sentences can appear much more coherent than others, given that the the required changes are pretty local (i.e., word substitution). On the other hand, style transfer models seem to have some difficulty in maintaining semantic context, so it seems that some of the gains in e.g. BLEU could be from CAST using the context simply as a bag of words to draw from in generating a style-transferred sentence. Instead of including context, the issue of semantic correctness could be fixed in other, simpler ways, however, such as making a more extractive single-sentence style transfer model. I think some further analysis is necessary to conclude that the context makes the generations more coherent, rather than helping the model in some other way.
Other general concerns/questions for the authors:
* The baselines do not appear to have learned even the single-sentence style transfer task that well. It appears that the authors used the default hyper parameters for other style transfer models (Hybrid Seq2Seq and ControlGen), but wouldn't it be better to choose hyper parameters for those models based on the evaluation criteria (as is done for CAST)? I know that some style transfer models can be quite sensitive to hyperparameters, so I would be surprised if the models worked optimally out of the box. I also feel that a strong unsupervised machine translation model like XLM (or the related style transfer model from "Multiple-attribute text style transfer") could do better than the provided baselines, as these models seem to work notably better than their predecessors.
* I have a general concern that CAST is quite complicated and has a lot of moving parts and parameters to tune. I don't think there are ablations on all components of the model (i.e., the style classifier). I also wouldn't be surprised if a simple state-of-the-art unsupervised or semi-supervised machine translation model like XLM outperformed CAST, without using any context. To add the use of the context, it seems possible to make a simple modification to an existing state-of-the-art transfer/translation model like XLM (adding the context to the input)
* Why are the style and coherence classifiers fixed during training? Wouldn't it be better to update the classifiers while training the rest of the model (so they doesn't get exploited)?
* Which of the evaluated models/baselines are trained using the style and/or coherence classifiers directly used for evaluation? For models that are not trained to optimize the style and/or coherence classifier predictions, I am not sure it is fair to compare those models to CAST on the style/coherence classifier accuracy, since CAST has been directly trained to perform well on that metric.
* Are the evaluation classifiers identical to those used to train CAST? At the least, it seems better to use a different style/coherence classifier (i.e., trained with a different random seed or other hyperparameters) at test time that the one used during training (i.e., in case CAST has overfit the style/coherence of generations to the classifiers used during training).
* In Eqn. 5, why is there a tanh activation immediately preceding the softmax? Wouldn't the tanh limit the range of the output softmax probabilities?
The proposed datasets do seem useful to the community, and the empirical results do support that CAST performs better than baselines - it would just be helpful to understand in what way the gains are coming from the additional context (to know if the task is an interesting task).