We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion. In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution. While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate. Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss. Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation. Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes. Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.
In this paper, the authors propose a probabilistic framework for unsupervised text style transfer. Given two non-parallel corpora X,Y in different domains, the authors introduce unobserved corpora \bar{X}, \bar{Y}. These are used as latent variables that control the generation of the observed data. To train models, the paper proposes to optimize the evidence lower bound of the log marginal likelihood. To facilitate training, multiple techniques are suggested, such as parameter sharing, some gradient approximations and initialization with a reconstruction objective. The approach is evaluated on five style transfer tasks, as well as unsupervised machine translation. Models are evaluated with multiple metrics, and generally obtain reasonably strong performance.
I lean towards the acceptance of the paper because the approach is fairly simple and elegant, while obtaining promising results. The connections to back-translation and language models are also potentially interesting. However, while the paper aims to suggest a principled approach to style transfer, using greedy samples biases the reconstruction objective, and as such the method does not really optimize the ELBO.
Casting style transfer as data completion is a straight-forward idea that doesn't introduce unnecessary or too simplistic assumptions. Optimizing the ELBO follows naturally, and can lead to more diverse outputs than the BT+NLL approach, which misses the negative entropy term. Reference BLEU scores on all tasks are competitive, and sometimes clearly better, with strong baselines.
Greedily sampling latent sequences during training should ideally be justified more carefully as it biases the objective function. In particular, an experimental comparison to stochastic sampling, which should more closely approximate the expectation, would be appreciated. Additionally, detailing the similarities and differences between the proposed approach and current UNMT techniques could be helpful to some readers.
Questions:
Could you present the validation and test evidence lower bounds? If so, how is sampling performed?
In footnote 2, you mention tuning the strength of the KL regularizer. As the KL can be decomposed into 2 terms (Eq. 5), would it be beneficial to control each term separately?