Abstract Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text data either rely on training an adversary (discriminator) that aims at making attribute values difficult to be inferred from the latent code or rely on minimising variational bounds of the mutual information between latent code and the value attribute. However, the available methods suffer of the impossibility to provide a fine-grained control of the degree (or force) of disentanglement. In contrast to adversarial methods, which are remarkably simple, although the adversary seems to be performing perfectly well during the training phase, after it is completed a fair amount of information about the undesired attribute still remains. This paper introduces a novel variational upper bound to the mutual information between an attribute and the latent code of an encoder. Our bound aims at controlling the approximation error via the Renyi's divergence, leading to both better disentangled representations and in particular, a precise control of the desirable degree of disentanglement than state-of-the-art methods proposed for textual data. Furthermore, it does not suffer from the degeneracy of other losses in multi-class scenarios. We show the superiority of this method on fair classification and on textual style transfer tasks. Additionally, we provide new insights illustrating various trade-offs in style transfer when attempting to learn disentangled representations and quality of the generated sentence. Learning disentangled representations hold a central place to build rich embeddings of high-dimensional data. For a representation to be disentangled implies that it factorizes some latent cause or causes of variation as formulated by (Bengio et al., 2013). For example, if there are two causes for the transformations in the data that do not generally happen together and are statistically distinguishable (e.g., factors occur independently), a maximally disentangled representation is expected to present a sparse structure that separates those causes. Disentangled representations have been shown to be useful for a large variety of data, such as video (Hsieh et al., 2018), image (Sanchez et al., 2019), text (John et al., 2018), audio (Hung et al., 2018), among others, and applied to many different tasks, e.g. , robust and fair classification (Elazar and Goldberg, 2018), visual reasoning (van Steenkiste et al., 2019), style transfer (Fu et al., 2017), conditional generation (Denton et al., 2017; Burgess et al., 2018), few shot learning (Kumar Verma et al., 2018), among others. In this work, we focus our attention on learning disentangled representations for text, as it remains overlooked by (John et al., 2018). Perhaps, one of the most popular applications of disentanglement in textual data is fair classification (Elazar and Goldberg, 2018; Barrett et al., 2019) and sentence generation tasks such as style transfer (John et al., 2018) or conditional sentence generation (Cheng et al., 2020b). For fair classification, perfectly disentangled latent representations can be used to ensure fairness as the decisions are taken based on representations which are statistically independent fromor at least carrying limited information aboutthe protected attributes. However, there exists a trade-offs between full disentangled representations and performances on the target task, as shown by (Feutry et al., 2018), among others. For sequence generation and in particular, for style transfer, learning disentangled representations aim at allowing an easier transfer of the desired style. To the best of our knowledge, a depth study of the relationship between disentangled representations based either on adversarial losses solely or on vCLUB  S and quality of the generated sentences remains overlooked. Most of the previous studies have been focusing on either trade-offs between metrics computed on the generated sentences (Tikhonov et al., 2019) or performance evaluation of the disentanglement as part of (or convoluted with) more complex modules. This enhances the need to provide a fair evaluation of disentanglement methods by isolating their individual contributions (Yamshchikov et al., 2019; Cheng et al., 2020b). Methods to enforce disentangled representations can be grouped into two different categories. The first category relies on an adversarial term in the training objective that aims at ensuring that sensitive attribute values ( e.g. race, sex, style) as statistically independent as possible from the encoded latent representation. Interestingly enough, several works (John et al., 2018; Elazar and Goldberg, 2018; Bao et al., 2019; Yi et al., 2020; Jain et al., 2019; Zhang et al., 2018; Hu et al., 2017), Elazar and Goldberg (2018) have recently shown that even though the adversary teacher seems to be performing remarkably well during training, after the training phase, a fair amount of information about the sensitive attributes still remains, and can be extracted from the encoded representation. The second category aim at minimising Mutual Information (MI) between encoded latent representation and the sensitive attribute values, i.e. , without resorting to an adversarial discriminator. MI acts as an universal measure of dependence since it captures non-linear and statistical dependencies of high orders between the involved quantities (Kin-ney and Atwal, 2014). However, estimating MI has been a long-standing challenge, in particular when dealing with high-dimensional data (Paninski, 2003; Pichler et al., 2020). Recent methods rely on variational upper bounds. For instance, (Cheng et al., 2020b) study vCLUB-S (Cheng et al., 2020a) for sentence generation tasks. Although this approach improves on previous state-of-the-art methods, it does not allow to fine-tuning of the desired degree of disentanglement, i.e., it enforces light or strong levels of disentanglement where only few features relevant to the input sentence remain (see Feutry et al. (2018) for further discussion). A novel objective to train disentangled representations from attributes. To overcome some of the limitations of both adversarial losses and vCLUB-S we derive a novel upper bound to the MI which aims at correcting the approximation error via either the Kullback-Leibler (Ali and Silvey, 1966) or Renyi (Rnyi et al., 1961) divergences. This correction terms appears to be a key feature to fine-tuning the degree of disentanglement compared to vCLUB-S . Applications and numerical results. First, we demonstrate that the aforementioned surrogate is better suited than the widely used adversarial losses as well as vCLUB-S as it can provide better disentangled textual representations while allowing fine-tuning of the desired degree of disentanglement . In particular, we show that our method offers a better accuracy versus disentanglement trade-offs for fair classification tasks. We additionally demonstrate that our surrogate outperforms both methods when learning disentangled representations for style transfer and conditional sentence generation while not suffering (or degenerating) when the number of classes is greater than two, which is an apparent limitation of adversarial training. By isolating the disentanglement module, we identify and report existing tradeoffs between different degree of disentanglement and quality of generated sentences. The later includes content preservation between input and generated sentences and accuracy on the generated style. We introduce notations, tasks, and closely related work. Consider a training set D = { ( x i , y i ) } ni =1 of n sentences x i  X paired with attribute values y i  Y  { 1 , . . . , |Y|} which indicates a discrete attribute to be disentangled from the resulting representations. We study the following scenarios: Disentangled representations. Learning disentangled representations consists in learning a model M : X  R d that maps feature inputs X to a vector of dimension d that retains as much as possible information of the original content from the input sentence but as little as possible about the undesired attribute Y . In this framework, content is defined as any relevant information present in X that does not depend on Y . Applications to binary fair classification. The task of fair classification through disentangled representations aims at building representations that are independent of selective discrete (sensitive) attributes ( e.g. , gender or race). This task consists in learning a model M : X  { 0 , 1 } that maps any input x to a label l  { 0 , 1 } . The goal of the learner is to build a predictor that assigns each x to either 0 or 1 oblivious of the protected attribute y . Recently, much progress has been made on devising appropriate means of fairness, e.g. , (Zemel et al., 2013; Zafar et al., 2017; Mohri et al., 2019). In particular, (Xie et al., 2017; Barrett et al., 2019; Elazar and Goldberg, 2018) approach the problem based on adversarial losses. More precisely, these approaches consist in learning an encoder that maps x into a representation vector h x , a critic C  c which attempts to predict y , and an output classifier f  d used to predict l based on the observed h x . The classifier is said to be fair if there is no statistical information about y that is present in h x (Xie et al., 2017; Elazar and Goldberg, 2018). Applications to conditional sentence generation. The task of conditional sentence generation consists in taking an input text containing specific stylistic properties to then generate a realistic (synthetic) text containing potentially different stylistic properties. It requests to learn a model M : X  Y  X that maps a pair of inputs ( x, y t ) to a sentence x g , where the outcome sentence should retain as much as possible of the original content from the input sentence while having (potentially a new) attribute y g . Proposed approaches to tackle textual style transfer (Zhang et al., 2020; Xu et al., 2019) can be divided into two main categories. The first category (Prabhumoye et al., 2018; Lample et al., 2018) uses cycle losses based on back translation (Wieting et al., 2017) to ensure that the content is preserved during the transformation. Whereas, the second category look to explicitly separate attributes from the content. This constraint is enforced using either adversarial training (Fu et al., 2017; Hu et al., 2017; Zhang et al., 2018; Yamshchikov et al., 2019) or MI minimisation using vCLUB-S (Cheng et al., 2020b). Traditional adversarial training is based on an encoder that aims to fool the adversary discriminator by removing attribute information from the content embedding (Elazar and Goldberg, 2018). As we will observe, the more the representations are disentangled the easier is to transfer the style but at the same time the less the content is preserved. In order to approach the sequence generation tasks, we build on the Style-embedding Model by (John et al., 2018) (StyleEmb) which uses adversarial losses introduced in prior work for these dedicated tasks. During the training phase, the input sentence is fed to a sentence encoder, namely f  e , while the input style is fed to a separated style encoder, namely f s e . During the inference phase, the desired stylepotentially different from the input styleis provided as input along with the input sentence. This section describes the proposed approach to learn disentangled representations. We first review MI along with the model overview and then, we derive the variational bound we will use, and discuss connections with adversarial losses. The MI is a key concept in information theory for measuring high-order statistical dependencies between random quantities. Given two random variables Z and Y , the MI is defined by I ( Z ; Y ) = EZY (cid:20) log p ZY ( Z, Y ) p Z ( Z ) p Y ( Y ) (cid:21) , (1) where p ZY is the joint probability density function (pdf) of the random variables ( Z, Y ) , with p Z and p Y representing the respective marginal pdfs. MI is related to entropy h ( Y ) and conditional entropy h ( Y | Z ) as follows: I ( Z ; Y ) = h ( Y )  h ( Y | Z ) . (2) Our models for fair classification and sequence generation share a similar structure. These rely on an encoder that takes as input a random sentence X and maps it to a random representation Z using a deep encoder denoted by f  e . Then, classification and sentence generation are performed using either a classifier or an auto-regressive decoder denoted by f  d . We aim at minimizing MI between the latent code represented by the Random Variable (RV) Z = f  e ( X ) and the desired attribute represented by the RV Y . The objective of interest L ( f  e ) is defined as: L ( f  e )  L down. where L down. represents a downstream specific (target task) loss and  is a meta-parameter that controls the sensitive trade-off between disentanglement ( i.e. , minimizing MI) and success in the downstream task ( i.e. , minimizing the target loss). In Sec. 5, we illustrate theses different trade-offs. Applications to fair classification and sentence generation. For fair classification, we follow standard practices and optimize the cross-entropy between prediction and ground-truth labels. In the sentence generation task L down. represents the negative log-likelihood between individual tokens. Estimating the MI is a long-standing challenge as the exact computation (Paninski, 2003) is only tractable for discrete variables, or for a limited family of problems where the underlying data-distribution satisfies smoothing properties, see recent work by (Pichler et al., 2020). Different from previous approaches leading to variational lower bounds (Belghazi et al., 2018; Hjelm et al., 2018; Oord et al., 2018), in this paper we derive an estimator based on a variational upper bound to the MI which control the approximation error based on the Kullback-Leibler and the Renyi divergences (Daudel et al., 2020). Theorem 1 (Variational upper bound on MI) Let ( Z, Y ) be an arbitrary pair of RVs with ( Z, Y )  p ZY according to some underlying pdf, and let q (cid:98) Y | Z be a conditional variational distribution on the attributes satisfying p ZY (cid:28) p Z  q (cid:98) Y | Z , i.e., absolutely continuous. Then, we have that I ( Z ; Y )  EY (cid:20)  log (cid:90) q (cid:98) Y | Z ( Y | z ) p Z ( z ) dz (cid:21) + EY Z (cid:104) log q (cid:98) Y | Z ( Y | Z ) (cid:105) + KL (cid:0) p ZY (cid:107) p Z  q (cid:98) Y | Z (cid:1) , (4) where KL (cid:0) p ZY (cid:107) p Z  q (cid:98) Y | Z (cid:1) denotes the KL divergence. Similarly, we have that for any  > 1 , I ( Z ; Y )  EY (cid:20)  log (cid:90) q (cid:98) Y | Z ( Y | z ) p Z ( z ) dz (cid:21) + EY Z (cid:104) log q (cid:98) Y | Z ( Y | Z ) (cid:105) + D  (cid:0) p ZY (cid:107) p Z  q (cid:98) Y | Z (cid:1) , (5) where (   1) D  (cid:0) p ZY (cid:107) p Z  q (cid:98) Y | Z (cid:1) = log EZY [ R   1 ( Z, Y )] denotes the Renyi divergence and R ( z, y ) = p Y | Z ( y | z ) q (cid:98) Y | Z ( y | z ) , for ( z, y )  Supp ( p ZY ) . Proof: The upper bound on H ( Y ) is a direct application of the the (Donsker and Varadhan, 1985) representation of KL divergence while the lower bound on H ( Y | Z ) follows from the monotonicity property of the function:  (cid:55) D  (cid:0) p ZY (cid:107) p Z  q (cid:98) Y | Z (cid:1) . Further details are relegated to Appendix A. Remark: It is worth to emphasise that the KL divergence in (4) and Renyi divergence in (5) control the approximation error between the exact entropy and its corresponding bound. From theoretical bounds to trainable surrogates to minimize MI: It is easy to check that the inequalities in (Eq. 4) and (Eq. 5) are tight provided that p ZY  p Z  q (cid:98) Y | Z almost surely for some adequate choice of the variational distribution. However, the evaluation of these bounds requires to obtain an estimate of the density-ratio R ( z, y ) . Density-ratio estimation has been widely studied in the literature (see (Sugiyama et al., 2012) and references therein) and confidence bounds has been reported by (Kpotufe, 2017) under some smoothing assumption on underlying data-distribution p ZY . In this work, we will estimate this ratio by using a critic C  R which is trained to differentiate between a balanced dataset of positive i.i.d samples coming from p ZY and negative i.i.d samples coming from q (cid:98) Y | Z  p Z . Then, for any pair ( z, y ) , the density-ratio can be estimated by R ( z, y )   ( C R ( z,y )) 1   ( C R ( z,y )) , where  (  ) indicates the sigmoid function and C  R ( z, y ) is the unnormalized output of the critic. It is worth to mention that after estimating this ratio, the previous upper bounds may not be strict bounds so we will refer them as surrogates. Adversarial approaches : In order to enhance our understanding of why the proposed approach based on the minimization of the MI using our variational upper bound in Th. 1 may lead to a better training objective than previous adversarial losses, we discuss below the explicit relationship between MI and cross-entropy loss. Let Y  Y denote a random attribute and let Z be a possibly high-dimensional representation that needs to be disentangled from Y . Then, I ( Z ; Y )  H ( Y )  EY Z (cid:104) log q (cid:98) Y | Z ( Y | Z ) (cid:105) = Const  CE ( (cid:98) Y | Z ) , (6) where CE ( (cid:98) Y | Z ) denotes the cross-entropy corresponding to the adversarial discriminator q (cid:98) Y | Z , noting that Y comes from an unknown distribution on which we have no influence H ( Y ) is an unknown constant, and using that the approximation error: KL (cid:0) q ZY (cid:107) q (cid:98) Y | Z  p Z (cid:1) = CE ( (cid:98) Y | Z )  H ( Y | Z ) . Eq. 6 shows that the cross-entropy loss leads to a lower bound (up to a constant) on the MI. Although the cross-entropy can lead to good estimates of the conditional entropy, the adversarial approaches for classification and sequence generation by (Barrett et al., 2019; John et al., 2018) which consists in maximizing the cross-entropy, induces a degeneracy (unbounded loss) as  increases in the underlying optimization problem. As we will observe in next section, our variational upper bound in Th. 1 can overcome this issue, in particular for |Y| > 2 . vCLUB-S : Different from our method, Cheng et al. (2020a) introduce I vCLUB which is an upper bound on MI defined by I vCLUB ( Y ; Z ) = EY Z [log p Y | Z ( Y | Z )]  EYEZ [log p Y | Z ( Y | Z )] . It would be worth to mention that this bound follows a similar approach to the previously introduced bound in (Feutry et al., 2018). Fair classification task. We follow the experimental protocol of (Elazar and Goldberg, 2018). The main task consists in predicting a binary label representing either the sentiment (positive/negative) or the mention. The mention task aims at predicting if a tweet is conversational. Here the considered protected attribute is the race. The dataset has been automatically constructed from DIAL corpus (Blodgett et al., 2016) which contained race annotations over 50 Million of tweets. Sentiment tweets are extracted using a list of predefined emo-jis and mentions are identified using @mentions tokens. The final dataset contains 160k tweets for the training and two splits of 10K tweets for validation and testing. Splits are balanced such that the random estimator is likely to achieve 50% accuracy. Style Transfer For our sentence generation task, we conduct experiments on three different datasets extracted from restaurant reviews in Yelp. The first dataset, referred to as SYelp, contains 444101, 63483, and 126670 labelled short reviews (at most 20 words) for train, validation, and test, respectively. For each review a binary label is assigned depending on its polarity. Following (Lample et al., 2018), we use a second version of Yelp, referred to as FYelp, with longer reviews (at most 70 words). It contains five coarse-grained restaurant category labels ( e.g. , Asian, American, Mexican, Bars and Dessert). The multi-category FYelp is used to access the generalization capabilities of our methods to a multi-class scenario. Efficiency measure of the disentanglement methods. (Barrett et al., 2019) report that offline classifiers (post training) outperform clearly adversarial discriminators. We will re-training a classifier on the latent representation learnt by the model and we will report its accuracy. Measure of performance within the fair classification task. In the fair classification task we aim at maximizing accuracy on the target task and so we will report the corresponding accuracy. Measure of performance within sentence generation tasks. Sentences generated by the model are expected to be fluent, to preserve the input content and to contain the desired style. For style transfer, the desired style is different from the input style while for conditional sentence generation, both input and output styles should be similar. Nevertheless, automatic evaluation of generative models for text is still an open problem. We measure the style of the output sentence by using a fastText classifier (Joulin et al., 2016b). For content preservation, we follow (John et al., 2018) and compute both: (i) the cosine measure between source and generated sentence embeddings, which are the concatenation of min, max, and mean of word embedding (sen-timent words removed), and (ii) the BLEU score between generated text and the input using SACRE-BLEU from (Post, 2018). Motivated by previous work, we evaluate the fluency of the language with the perplexity given by a GPT-2 (Radford et al., 2019) pretrained model performing fine-tuning on the training corpus. We choose to report the log-perplexity since we believe it can better reflects the uncertainty of the language model (a small variation in the model loss would induce a large change in the perplexity due to the exponential term). Besides the automatic evaluation, we further test our disentangled representation effectiveness by human evaluation results are presented in Tab. 1. vCLUB-S , KL refers to a model trained using the vCLUB-S and KL surrogate (see Eq. 14) respectively; and D  refers to a model trained based on the  -Renyi surrogate (Eq. 15), for   { 1 . 3 , 1 . 5 , 1 . 8 } . In this section, we present our results on the fair classification and binary sequence generation tasks, see Ssec. 5.1 and Ssec. 5.2, respectively. We additionally show that our variational surrogates to the MIcontrarily to adversarial lossesdo not suffer in multi-class scenarios (see Ssec. 5.3). Upper bound on performances. We first examine how much of the protected attribute we can be recovered from an unfair classifier ( i.e. , trained without adversarial loss) and how well does such classifier perform. Results are reported in Fig. 1. We observe that we achieve similar scores than the ones reported in previous studies (Barrett et al., 2019; Elazar and Goldberg, 2018). This experiment shows that, when training to solve the main task, the classifier learns information about the protected attribute, i.e. , the attacker's accuracy is better than random guessing. In the following, we compare the different proposed methods to disentangle representations and obtain a fairer classifier. Methods comparisons. Fig. 1 shows the results of the different models and illustrates the trade-offs between disentangled representations and the target task accuracy. Results are reported on the testset for both sentiment and mention tasks when race is the protected. We observe that the classifier trained with an adversarial loss degenerates for  > 5 since the adversarial term in Eq. 3 is influencing much the global gradient than the downstream term ( i.e. , cross-entropy loss between predicted and golden distribution). Remarkably, both models trained to minimize either the KL or the Renyi surrogate do not suffer much from the aforementioned multiclass problem. For both tasks, we observe that the KL and the Renyi surrogates can offer better disentangled representations than those induced by adversarial approaches. In this task, both the KL and Renyi achieve perfect disentangled representations ( i.e. , random guessing accuracy on protected attributes) with a 5% drop in the accuracy of the target task, when perfectly masking the protected attributes. As a matter of fact, we observe that vCLUB-S provides only two regimes: either a light protection (attacker accuracy around 60%), with almost no loss in task accuracy (  < 1 ), or a strong protection (attacker accuracy around 50%), where a few features relevant to the target task remain. 1 On the sentiment task, we can draw similar conclusions. However, the Renyi's surrogate achieves slightly better-disentangled representations. Overall, we can observe that our proposed surrogate enables good control of the degree of disentangling. Additionally, we do not observe a degenerated behaviouras it is the case with adversarial losseswhen  increases. Furthermore, our surrogate allows simultaneously better disentangled representations while preserving the accuracy of the target task. In the previous section, we have shown that the proposed surrogates do not suffer from limitations of adversarial losses and allow to achieve better disentangled representations than existing methods relying on vCLUB-S . Disentanglement modules are a core block for a large number of both style transfer and conditional sentence generation algorithms (Tikhonov et al., 2019; Yamshchikov et al., 2019; Fu et al., 2017) that place explicit constraints to force disentangled representations. First, we assess the disentanglement quality and the control over desired level of disentanglement while changing the downstream term, which for the sentence generation task is the cross-entropy loss on individual token. Then, we exhibit the existing trade-offs between quality of generated sentences, measured by the metric introduced in Ssec. 4.2, and the resulting degree of disentanglement. The results are presented for SYelp 5.2.1 Evaluating disentanglement Fig. 2a shows the adversary accuracy of the different methods as a function of  . Similarly to the fair classification task, a fair amount of information can be recovered from the embedding learnt with adversarial loss. In addition, we observe a clear degradation of its performance for values  > 1 . In this setting, the Renyi surrogates achieves consistently better results in terms of disentanglement than the one minimizing the KL surrogate. The curve for Renyi's surrogates shows that exploring different values of  allows good control of the 1 This phenomenon is also reported in (Feutry et al., 2018) on a picture anonymization task. (d) Figure 1: Numerical results on fair classification. Trade-offs between target task and attacker accuracy are reported in Fig. 1a, Fig. 1b for mention task, and Fig. 1c, Fig. 1d for sentiment task. For low values of  some points coincide. As  increases the level of disentanglement increases and the proposed methods using both KL ( KL ) and Reny divergences ( D  ) clearly offer better control than existing methods. disentanglement degree. Renyi surrogate generalizes well for sentence generation. Similarly to the fairness task vCLUB-S only offers two regimes: "light" disentanglement with very little polarity transfer and "strong" disentanglement. The quality of generated sentences are evaluated using the fluency (see Fig. 3c ), the content preservation (see Fig. 3a), additional results using a cosine similarity are given in Appendix D, and polarity accuracy (see Fig. 3b ). For style transfer, and for all models, we observe trade-offs between disentanglement and content preservation (measured by BLEU) and between fluency and disentanglement. Learning disentangled representations leads to poorer content preservation. As a matter of fact, similar conclusions can be drawn while measuring content with the cosine similarity (see Appendix D). For polarity accuracy, in non-degenerated cases (see below), we observe that the model is able to better transfer the sentiment in presence of disentangled representations. Transferring style is easier with disentangled representations, however there is no free lunch here since disentangling also removes important information about the content . It is worth noting that even in the "strong" disentanglement regime vCLUB-S struggles to transfer the polarity (accuracy of 40% for   { 1 , 2 , 10 , 15 } ) where other models reach 80%. It is worth noting that similar conclusions hold for two different sentence generation tasks: style transfer and conditional generation, which tends to validate the current line of work that formulates text generation as generic text-to-text (Raffel et al., 2019). Quality of generated sentences. Examples of generated sentences are given in Tab. 2 , providing qualitative examples that illustrate the previously observed trade-offs. The adversarial loss degenerates for values   5 and a stuttering phenomenon appears (Holtzman et al., 2019). Tab. 1 gathers results of human evaluation and show that our surrogates can better disentangle style while preserving more content than available methods. In Fig. 2b we report the adversary accuracy of our different methods for the values of  using FYelp dataset with category label. In the binary setting for   1 , models using adversarial loss can learn disentangled representations while in the multi-class setting, the adversarial loss degenerates for small values of  ( i.e sentences are no longer fluent as shown by the increase in perplexity in Fig. 4c). Minimizing MI based on our surrogates seems to mitigate the problem and offer a better control of the disentanglement degree for various values of  than vCLUB  S . Further results are gathered in Appendix G. 6 Summary and Concluding Remarks We devised a new alternative method to adversarial losses capable of learning disentangled textual representation. Our method does not require adversarial training and hence, it does not suffer in presence of multi-class setups. A key feature of this method is to account for the approximation error incurred when bounding the mutual information. Experiments show better trade-offs than both adversarial training and vCLUB-S on two fair classification tasks and demonstrate the efficiency to learn disentangled representations for sequence generation. As a matter of fact, there is no free-lunch for sentence generation tasks: although transferring style is easier with disentangled representations, it also removes important information about the content . The proposed method can replace the adversary in any kind of algorithms (Tikhonov et al., 2019; Fu et al., 2017) with no modifications. Future work includes testing with other type of labels such as dialog act (Chapuis et al., 2020; Colombo et al., 2020), emotions (Witon et al., 2018), opinion (Gar-cia et al., 2019) or speaker's stance and confidence (Dinkar et al., 2020). Since it allows more fine-grained control over the amount of disentanglement, we expect it to be easier to tune when combined with more complex models. The authors would like to thanks Georg Pichler for the thorough reading. The work of Prof. Pablo Piantanida was supported by the European Commis-sion's Marie Sklodowska-Curie Actions (MSCA), through the Marie Sklodowska-Curie IF (H2020-MSCAIF-2017-EF-797805). The PhD of Pierre is fully founded by IBM GBS France in collaboration with Telecom Paris.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain method abstain abstain objective objective other objective objective objective objective objective objective objective abstain objective objective objective result objective method objective abstain other method other other other other other other other other other other other other other other other other other other abstain method other other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other other abstain abstain abstain abstain abstain method abstain abstain method method abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other