Abstract When working with textual data, a natural application of disentangled representations is fair classification where the goal is to make predictions without being biased (or influenced) by sensitive attributes that may be present in the data (e.g., age, gender or race). Dominant approaches to disentangle a sensitive attribute from textual representations rely on learning simultaneously a penalization term that involves either an adversarial loss (e.g., a discriminator) or an information measure (e.g., mutual information). However, these methods require the training of a deep neural network with several parameter updates for each update of the representation model. As a matter of fact, the resulting nested optimization loop is both time consuming, adding complexity to the optimization dynamic, and requires a fine hyperparame-ter selection (e.g., learning rates, architecture). In this work, we introduce a family of regularizers for learning disentangled representations that do not require training. These regularizers are based on statistical measures of similarity between the conditional probability distributions with respect to the sensitive attributes. Our novel regularizers do not require additional training, are faster and do not involve additional tuning while achieving better results both when combined with pretrained and randomly initialized text encoders. As natural language processing (NLP) systems are taken up in an ever wider array of sectors (e.g., legal system (Dale, 2019), insurance (Ly et al., 2020), education (Litman, 2016), healthcare (Basyal et al., 2020)), there are growing concerns about the harmful potential of bias in such systems (Leidner and Plachouras, 2017). Recently, a large body of research aims at analyzing, understanding and addressing bias in various applications of NLP including language modelling (Liang et al., 2021), machine translation (Stanovsky et al., 2019), toxic-40 20 0 20 40 20 10 0 10 20 30 Original T=0 male female Fig. 1: PCA followed by a T-SNE projection of BERT embeddings of the sentences of DIAL corpus after T=0,10,1000 iterations of our framework (based on Sinkhorn divergence). Colors display the sensitive (i.e., binary gender) attribute. ity detection (Dixon et al., 2018) and classification (Elazar and Goldberg, 2018). In NLP, current systems often rely on learning continuous embedding of the input text. Thus, it is crucial to ensure that the learnt continuous representations do not exhibit bias that could cause representational harms (Blodgett et al., 2020; Barocas et al., 2017), i.e., representations less favourable to specific social groups. One way to prevent the aforementioned phenomenon is to enforce disentangled representations, i.e., representations that are independent of a sensitive attribute (see Fig. 1 for a visualization of different degrees of disentangled representations). Learning disentangled representations has received a growing interest as it has been shown to be useful for a wide variety of tasks (e.g., style transfer (Fu et al., 2017), few shot learning (Karn et al., 2021), fair classification (Colombo et al., 2021d)). For text, the dominant approaches to learn such representations can be divided into two classes. The first one, relies on an adversary that is trained to recover the discrete sensitive attribute from the latent representation of the input (Xie et al., 2017). However, as pointed out by Barrett et al. (2019), even though the adversary seems to do a perfect job during training, a fair amount of the sensitive information can be recovered from the latent representation when training a new adversary from scratch. The second line of research involves a regularizer that is a train-2614 able surrogate of the mutual information (MI) (e.g., CLUB (Cheng et al., 2020a), MIReny (Colombo et al., 2021d), KNIFE (Pichler et al., 2020), MINE (Belghazi et al., 2018; Colombo et al., 2021b)) and achieves higher degrees of disentanglement. However, as highlighted by recent works (McAllester and Stratos, 2020; Song and Ermon, 2019), these estimators are hard to use in practice and the optimization procedure (see App. D.4) involves several updates of the regularizer parameters at each update of the representation model. As a consequence, these procedures are both time consuming and involve extra hyperparameters (e.g., optimizer learning rates, architecture, number of updates of the nested loop) that need to be carefully selected which is often not such an easy task. Contributions. In this work, we focus our attention on learning to disentangle textual representations from a discrete attribute. Our method relies on a novel family of regularizers based on discrepancy measures. We evaluate both the disentanglement and representation quality on fair text classification. Formally, our contribution is two-fold: (1) A novel formulation of the problem of learning disentangled representations. Different from previous workseither minimizing a surrogate of MI or training an adversarywe propose to minimize a statistical measure of similarity between the underlying probability distributions conditioned to the sensitive attributes. This novel formulation allows us to derive new regularizers with convenient properties: (i) not requiring additional learnable parameters; (ii) alleviating computation burden; and (iii) simplifying the optimization dynamic. (2) Applications and numerical results. We carefully evaluate our new framework on four different settings coming from two different datasets. We strengthen the experimental protocol of previous works (Colombo et al., 2021d; Ravfogel et al., 2020) and test our approach both on randomly initialized encoder (using RNN-based encoder) and during fine-tuning of deep contextualized pretrained representations 1 . Our experiments are conducted on four different main/sensitive attribute pairs and involve the training of over 280 deep neural networks. Our findings show that: (i) disentanglement methods behave differently when applied to randomly initialized or to deep contextualized pretrained encoder; and (ii) our framework offers a 1 Previous works (e.g., (Ravfogel et al., 2020)) do not fine-tune the pretrained encoder when testing their methods. better accuracy/disentanglement trade-off than existing methods (i.e., relying on an adversary or on a MI estimator) while being faster and easier to train. Model, data and code are available at https:// github.com/PierreColombo/TORNADO . Considering a tuple ( X, S ) where X is a random variable (r.v.) defined on the space of text X and S is a binary r.v. which corresponds to a sensitive attribute. Learning disentangled representations aims at learning the parameter  of the encoder f  : X  Z  R d which maps X to a latent representation Z = f  ( X )  R d , where d  N  corresponds to the dimension of the embedding space. The goal is that Z retains as much useful information from X while being oblivious of S . Among the numerous possible applications for disentangled representations, we choose to focus on fair classification as it is a natural task to define the aforementioned useful information. In the fair classification task, we assume access to Y , a binary r.v., which corresponds to the main label/attribute. In order to learn disentangled representations for fair classification, we follow previous works (Beu-tel et al., 2017; Cheng et al., 2020b) and we will be minimizing the loss L ( , ,  ) , which is defined as follows: CE (cid:0) C  ( f  ( X )) , Y (cid:1) (cid:124) (cid:123)(cid:122) (cid:125) target task +   R (cid:0) f  ( X ) , S ;  (cid:1) (cid:124) (cid:123)(cid:122) (cid:125) regularizer , (1) where C  : Z  Y refers to the main classifier;  to its learnable parameters; CE to the cross-entropy loss; R denotes the disentanglement regularizer;  its parameters and  controls the trade-off between disentanglement and success in the classification task. We next review the two main methods that currently exist for learning textual disentangled representations: adversarial-based and MI-based . In the context of disentangled representation learning, a popular method is to rely on adding an adversary to the encoder (e.g., texts (Coavoux et al., 2018), images (Xie et al., 2017), categorical data (Beutel et al., 2017)). This adversary is competing against the encoder trying to learn the main task objective. In this line of work, R ( f  ( X ) , S ;  ) =  CE ( C  ( f  ( X )) , S ) where C  : Z  S refers to the adversarial classifier that is trained to minimize CE ( C  ( f  ( X )) , S ) . Denoting by PZ | S =0 2615 and PZ | S =1 the probability distribution of the conditional r.v. Z | S = 0 and Z | S = 1 , respectively, these works build on the fact that if PZ | S =0 and PZ | S =1 are different, the optimal adversary will be able to recover sensitive information from the latent code Z . Although adversaries have achieved impressive results in many applications when applied to attribute removal, still a fair amount of information may remain in the latent representation (Lample et al., 2018). To better protect sensitive information, the second class of methods involves direct mutual information minimization. MI lies at the heart of information theory and measures statistical dependencies between two random variables Z and S and find many applications in machine learning (Boudiaf et al., 2020b,a, 2021). The MI is a non-negative quantity that is 0 if and only if Z and S are independent and is defined as follows: I ( Z ; S ) = KL ( PZS  PZ  PS ) , (2) where the joint probability distribution of ( Z, S ) is denoted by PZS ; marginals of Z and S are denoted by PZ and PS respectively; and KL stands for the Kullback-Leibler divergence. Although computing the MI is challenging (Paninski, 2003; Pichler et al., 2020), a plethora of recent works devise new lower (Belghazi et al., 2018; Oord et al., 2018) and upper bounds (Cheng et al., 2020a; Colombo et al., 2021d)  I  ( f  ( X ); S ) where  denotes the trainable parameters of the surrogate of the MI. In that case, R ( f  ( X ) , S ;  ) =  I  ( f  ( X ); S ) . These methods build on the observation that if I ( Z ; X ) > 0 then PZ | S =0 and PZ | S =1 are different and information about the sensitive label S remains in Z . Interestingly, these approaches achieve better results than adversarial training on various NLP tasks (Cheng et al., 2020b) but involve the use of additional (auxiliary) neural networks. The aforementioned methods involve the use of extra parameters (i.e.,  ) in the regularizer. As the regularizer computes a quantity based on the representation given by the encoder with parameter  , any modification of  requires an adaptation of the parameter of R (i.e.,  ). In practice, this adaptation is performed using gradient descent-based algorithms and requires several gradient updates. Thus, a nested loop (see App. D.4) is needed. Additional optimization parameters and the nested loop both induce additional complexity and require a fine-tuning which makes these procedures hard to be used on large-scale datasets. To alleviate these issues, the next section describes a parameter-free framework to get rid of the  parameter present in R . This section describes our approach to learn disentangled representations. We first introduce the main idea and provide an algorithm to implement the general loss. We next describe the four similarity measures proposed in this approach. As detailed in Section 2, existent methods generally rely on the use of neural networks either in the form of an adversarial regularizer or to compute upper/lower bounds of the MI between the embedding Z = f  ( X ) and the sensitive attribute S . Motivated by reducing the computational and complexity load, we aim at providing regularizers that are light and easy to tune. To this end, we need to get rid of the nested optimization loop, which is both time consuming and hard to tune in practice since the regularizer contains a large number of parameters (e.g., neural networks) that need to be trained by gradient descent. Contrarily to previous works in the literature, and following the intuitive idea that PZ | S =0 and PZ | S =1 should be as close as possible, we introduce similarity measures between PZ | S =0 and PZ | S =1 to build a regularizer R . It is worth noting that the similarity measures do not require any additional learnable parameters. For the sake of clarity, in the reminder of the paper we define P i  PZ | S = i and Z i  f  ( X | S = i ) for i  { 0 , 1 } . Given a similarity measure defined as SM : M 1+ ( Z )  M 1+ ( Z )  R + where M 1+ ( Z ) denotes the space of probability distributions on Z , we propose to regularize the downstream task by SM ( P 0 , P 1 ) . Precisely, the optimization problem boils down to the following objective: L ( ,  ) = CE ( C  ( f  ( X )) , Y ) (cid:124) (cid:123)(cid:122) (cid:125) target task +   SM ( P 0 , P 1 ) (cid:124) (cid:123)(cid:122) (cid:125) regularizer . The proposed statistical measures of similarity, detailed in Section 3.2, have explicit and simple formulas. It follows that the use of neural networks is no longer necessary in the regularizer term which 2616 reduces drastically the complexity of the resulting learning problem. The disentanglement can be controlled by selecting appropriately the measure SM. For the sake of place, the algorithm we propose to solve (3) is deferred to the App. B. 3.2 Measure of Similarity between Distributions In this work, we choose to focus on four different (dis-) similarity functions ranging from the most popular in machine learning such as the Maximum Mean Discrepancy measure (MMD) and the Sinkhorn divergence (SD) to standard statistical discrepancies such as the Jeffrey divergence (J) and the Fisher-Rao distance (FR). Let k : Z  Z  R be a kernel and H its corresponding Reproducing Kernel Hilbert Space with inner product ., . H and norm  . H . Denote by BH = { f |  f  H  1 } the unit ball of H . The Maximum Mean Discrepancy (MMD) (Gretton et al., 2007) between the two conditional distributions P 0 , P 1  M 1+ ( Z ) associated with the kernel k , is defined as: MMD ( P 0 , P 1 ) = sup  B H (cid:12)(cid:12) EP 0 [( Z 0 )]  EP 1 [( Z 1 )] (cid:12)(cid:12) = EP 0  P 0 [ k ( Z 0 , Z  0 )] + EP 1  P 1 [ k ( Z 1 , Z  1 )]  2 EP 0  P 1 [ k ( Z 0 , Z 1 )] . The MMD can be estimated with a quadratic computational complexity O ( n 2 ) where n is the sample size. In this paper, MMD is computed using the Gaussian kernel k : ( z 0 , z 1 ) (cid:55) exp(  z 0  z 1  2 / 2  2 ) , where    is the usual euclidean norm. The Wasserstein distance aims at comparing two probability distributions through the resolution of the Monge-Kantorovich mass transportation problem (see e.g. Villani (2003); Peyr and Cuturi (2019)): W p ( P 0 , P 1 ) = min   U ( P 0 , P 1 ) (cid:90) ZZ  z 0  z 1  p d ( z 0 , z 1 ) , (4) where U ( P 0 , P 1 ) = {   M 1 + ( Z  Z ) : (cid:82)  ( z 0 , z 1 ) dy = P 0 ( z 0 ); (cid:82)  ( z 0 , z 1 ) dx = P 1 ( z 1 ) } is the set of joint probability distributions with marginals P 0 and P 1 . For the sake of clarity, the power p in W p is omitted in the remainder of the paper. When P 0 and P 1 are discrete measures, (4) is a linear problem and can be solved with a supercubic complexity O ( n 3 log( n )) , where n denotes the sample size. To overcome this computational drawback, Cuturi et al. (2013) added an entropic regularization term to the transport cost to obtain a strongly convex problem solvable using the Sinkhorn-Knopp algorithm (Sinkhorn, 1964) leading to a computational cost of O ( n 2 ) . The bias introduced by the regularization term, i.e., the quantity is not longer zero when comparing to the same probability distribution, have been corrected by Genevay et al. (2019) leading to the known Sinkhorn Divergence (SD) defined as: SD  ( P 0 , P 1 ) = W  ( P 0 , P 1 )  1 2 1 (cid:88) i =0 W  ( P i , P i ) , where W  ( P 0 , P 1 ) is equal to min   U ( P 0 , P 1 ) (cid:90) ZZ  z 0  z 1  p d ( z 0 , z 1 ) +  H (  ) , with H (  ) = (cid:82)  ( z 0 , z 1 ) log(  ( z 0 , z 1 )) dz 0 dz 1 . The Fisher-Rao distance (FR) (Rao, 1945) is a Riemannian metric defined on the space of parametric distributions relying on the Fisher information. The Fisher information matrix provides a natural Riemannian structure (Amari, 2012). It is known to be more accurate than popular divergence measures (Costa et al., 2015). Let M 1+ ( Z , P ) be the family of parametric distributions with the parameter space P  R d . The FR distance is defined as the geodesic distance 2 between elements (i.e., probability measures) on the manifold M 1+ ( Z , P ) . Parametrizing P 0 , P 1 by parameters p 1 , p 2  P , respectively, such that P p 0 0  P 0 and P p 1 1  P 1 , the FR distance between P p 0 0 and P p 1 1 is defined as: FR ( P p 0 0 , P p 1 1 ) = min  (cid:90) | (cid:113)   ( t )  G ( p 0 , p 1 )  ( t ) | dt (5) where  ( t ) is the curve connecting p 0 and p 1 in the parameter space P ; and G ( p 0 , p 1 ) is the Fisher information matrix of ( p 0 , p 1 ) . In general, the optimization problem of (5) can be solved using the well-known Euler-Lagrange differential equations leading to computational difficulties. Atkinson and Mitchell (1981) have provided computable closed-form for specific families of distributions such as 2 The geodesic is the curve that provides the shortest length. Multivariate Gaussian with diagonal covariance matrix. Under this assumption, the parameters p 0 and p 1 are defined by p i,j = (  i,j ,  i,j )  R 2 for i  { 0 , 1 } and 1  j  d with  i  R d the mean vector and Diag (  i ) the diagonal covariance matrix of P i where  i is the variance vector. The resulting FR metric admits the following closed-form (see e.g. Pinele et al. (2020): FR ( P p 0 0 , P p 1 1 ) = (cid:118)(cid:117)(cid:117)(cid:116) d (cid:88) j =1 [ d FR ( p 0 ,j , p 1 ,j )] 2 , where d FR ( p 0 ,j , p 1 ,j ) is the univariate Fisher-Rao detailed in the App. The Jeffrey divergence (J) is a symmetric version of the Kullback-Leibler (KL) divergence and measures the similarity between two probability distributions. Formally, it is defined as follow: J ( P 0 , P 1 ) = 1 2 (cid:2) KL ( P 0  P 1 ) + KL ( P 1  P 0 ) (cid:3) . Computing the KL ( P 0  P 1 ) either requires to have knowledge of P 0 and P 1 , or to have knowledge about the density ratio (Rubenstein et al., 2019). Without any further assumption on P 0 , P 1 or the density ratio, the resulting inference problem is known to be provably hard (Nguyen et al., 2010). Although previous works have addressed the estimation problem without making assumptions on P 0 and P 1 (Oord et al., 2018; Hjelm et al., 2018; Belghazi et al., 2018), these methods often involve additional parameters (e.g., neural networks (Song and Ermon, 2019), kernels (McAllester and Stratos, 2020)), require additional tuning (Hershey and Olsen, 2007), and are time expensive. Motivated by speed, simplicity and to allow for fair comparison with FR, for this specific divergence, we choose to make the assumption that P 0 and P 1 are multivariate Gaussian distributions with mean vector  0 and  1 and diagonal covariance matrices:  0 and  1 . Thus, KL ( P 0 , P 1 ) boils down to: log |  0 | |  1 |  d + Tr (  1 0  1 )+(  0   1 ) T   1 0 (  0   1 ) , where Tr (  1 0  1 ) is the trace of   1 0  1 . Remark. FR and J are computed under the multivariate Gaussian with diagonal covariance matrix assumption. In this case, the Sinkhorn approximation is not needed as (4) can be efficiently computed thanks to the following closed-form: W( P 0 , P 1 )=   0   1  2 +Tr (cid:16)  0 + 1  2( 0  1 ) 1 / 2 (cid:17) Remark. Quantities defined in this section are replaced by their empirical estimate. Due to space constraints, the formula are described in App. A.2. In this section, we describe the datasets, metrics, encoder and baseline choices. Additional experimental details can be found in App. D. For fair comparison, all models were re-implemented. To ensure backward comparison with previous works, we choose to rely on the DIAL (Blodgett et al., 2016) and the PAN (Rangel et al., 2014) datasets. For both, main task labels ( Y ) and sensitive labels ( S ) are binary, balanced and splits follow (Barrett et al., 2019). Random guessing is expected to achieve near 50% of accuracy. The DIAL corpus has been automatically built from tweets and the main task is either polarity 3 or mention prediction. The sensitive attribute is related to race (i.e., non-Hispanic blacks and non-Hispanic whites) which is obtained using the author geo-location and the words used in the tweet. The PAN corpus is also composed of tweets and the main task is to predict a mention label. The sensitive attribute is obtained through a manual process and annotations contain the age and gender information from 436 Twitter users. For the choice of the evaluation metrics, we follow the experimental setting of Colombo et al. (2021d); Elazar and Goldberg (2018); Coavoux et al. (2018). To measure the success of the main task, we report the classification accuracy. To measure the degree of disentanglement of the latent representation we train from scratch an adversary to predict the sensitive labels from the latent representation. In this framework, a perfect model would achieve a high main task accuracy (i.e., near 100%) and a low (i.e., near 50%) accuracy as given by the adversary prediction on the sensitive labels. Following Colombo 3 Polarity or emotion have been widely studied in the NLP community (Jalalzai et al., 2020; Colombo et al., 2019) 2618 RNN BERT Dat. Tab. 1: Results on the fair classification task: the main task (higher is better) accuracy corresponds to the column with Y (  ) and S (  ) denotes the sensitive task accuracy (lower is better). CE refers to a classifier trained with CE loss solely (  = 0 in (1)). et al. (2021d), we also report the disentanglement dynamic following variations of  and train a different model for each   [0 . 001 , 0 . 01 , 0 . 1 , 1 , 10] . Choice of the encoder. Previous works that aim at learning disentangled representations either focus on randomly initialized RNN-encoders (Colombo et al., 2021d; Elazar and Goldberg, 2018; Coavoux et al., 2018) or only use pretrained representations as a feature extractor (Ravfogel et al., 2020). In this work, we choose to fine-tune BERT during training as we believe it to be a more realistic setting. Choice of the baseline models. We choose to compare our methods against adversarial training from Elazar and Goldberg (2018); Coavoux et al. (2018) (model named ADV) and the recently MI bound introduced in (Colombo et al., 2021d) (named MI) which has been shown to be more controllable than previous MI-based estimators. In this section, we gather experimental results for fair classification task. We study our framework when working either with RNN or BERT encoders. The parameter  (see (3)) controls the trade-off between success on the main task and disentanglement for all models. Y are tightly entangled. By comparing Fig. 2 and Fig. 3, we notice that the race label (main task) is easier to disentangled from the sentiment compared to the mention. Randomly initialized RNN encoders. To allow a fair comparison with previous works, we start by testing our framework with RNN encoders on the DIAL dataset. Results are depicted in Fig. 2. It is worth mentioning that we are able to observe a similar phenomenon that the one reported in Colombo et al. (2021d). More specifically, we observe: (i) the adversary degenerates for  = 10 and does not allow to reach perfectly disentangled representations nor to control the desirable degree of disentanglement; (ii) the MI allows better control over the desirable degree of disentanglement and achieves better-disentangled representations at a reduced cost on the main task accuracy. Fig. 2 shows that the encoder trained using the statistical measures of similarityboth with and without the multivariate Gaussian assumptionare able to learn disentangled representations. We can also remark that our losses follow an expected behaviour: when  increases, more weight is given to the regularizer, the sensitive task accuracy decreases, thus the representations are more disentangled according to the probing-classifier. Overall, we observe that the W regularizer is the best performer with optimal performance for  = 1 on both attributes. On the other hand, we observe that FR and J divergence are useful to learn to disentangle the representations but disentangling using these similarity measures comes with a greater cost as compared to W. Both MMD and SD also perform well 4 and are able to learn disentangled representations with little cost on the main task performance. However, on DIAL , they are not able to learn perfectly disentangled representations. Similar conclusions can be drawn on PAN and results are reported in App. C.1. BERT encoder. Results of the experiment conducted with BERT encoder are reported in Fig. 3. As expected, we notice that on both tasks the main and the sensitive task accuracy for small values of  is higher than when working with RNN encoders. When training a classifier without disentanglement constraints (i.e., case  = 0 in (1)), which corresponds to the dash lines in Fig. 2 and Fig. 3, we observe that BERT encoder naturally preserves more sensitive information (i.e., measured 4 For both losses when  > 10 we did not remark any consistent improvements. 2619 by the accuracy of the adversary) than randomly initialized encoder. Contrarily to what is usually undertaken in previous works (e.g., Ravfogel et al. (2020)), we allow the gradient to flow in BERT encoder while preforming fine-tuning. We observe a different behavior when compared to previous experiments. Our losses under the Multivariate diagonal Gaussian assumption (i.e., W, J, FR ) can only disentangle the representations at a high cost on the main task (i.e., perfect disentanglement corresponds to performance on the main task close to a random classifier). When training the encoder with either SD or MMD, we are able to learn disentangled representations with a limited cost on the main task accuracy:  = 0 . 1 achieves good disentanglement with less than 3% of loss in the main task accuracy. The methods allow little control over the degree of disentanglement and there is a steep transition between light protection with no loss on the main task accuracy and strong protection with discriminative features destruction. Fig. 2: Results on DIAL with RNN. Dash lines correspond to model trained with CE loss solely (i.e., case  = 0 in (1)). Figures on the left are dedicated to the mention attribute while the one on the rights reports results on the Sentiment attribute. The main task consists in predicting Y thus higher is better. The sensitive task accuracy is obtained by training a classifier to S on the final representation thus an ideal model would reach 50% of accuracy. Takeaways. Our new framework relying on statistical Measures of Similarity introduces powerful methods to learn disentangled representations. When working with randomly initialized RNN encoders to learn disentangled representation, we advise relying on W. Whereas in presence of pretrained encoders (i.e., BERT), we observe a very different behavior 5 and recommend using SD. Fig. 3: Results on DIAL for mention (left) and sentiment (right) attribute using a pretrained BERT. We report in Table 2 the training time and the number of parameters of each method. The reduced number of parameters brought by our method is marginal, however getting rid of these parameters is crucial. Indeed, they require a nested loop and require a fined selection of the hyperparameters which complexify the global system dynamic. Takeaways. Contrarily to MI or Adversarial based regularizer that are difficult (or even prohibitive) to be implemented on large-scale datasets, our framework is simpler and consistently faster which makes it a better candidate when working with large-scale datasets. Results presented in Section 5.1 have shown a different behaviour for RNN and BERT based encoders and, for different measures of similarity. Tab. 2: Speed and number of model parameters (given in thousand) when working with DIAL . The runtime for 1 gradient update (denoted 1 upd.) or for 1 epoch is given for a batch of 64 when running our models on a single NVIDIA-V100. The relative improvements (in %) are given with respect to the MI model, which is our strongest baseline. Here, we aim at understanding of this phenomena. In the previous section, we examine the change the measures during the training. Takeaways. When using a RNN encoder, the system is able to maximize the main task accuracy while jointly minimizing most of the similarity measures. For BERT where the model is more complex, for measures relying on the diagonal gaussian multivariate assumption either the disentanglement plateau (e.g., FR or J) or the system fails to learn discriminative features and perform poorly on the main task (e.g., W). When combined with BERT both SD and MMD can achieve high main task accuracy while protecting the sensitive attribute. In this experiment, we investigate how predictive of the disentanglement is each similarity measure, i.e., does a lower value of similarity measure indicates better disentangled representations? We gather for both the mention and sentiment attribute 5 checkpoints per model (i.e., each regularizer and each value of  corresponds to one model). For each RNN model, we select one checkpoint after 5k , 10k , 15k , 20k , 25k gradient updates, and for BERT we select one checkpoint after 2k , 4k , 6k , 8k , 10k gradient updates to obtain the same number of models. For each type of loss, we ended with 50 models. For each model and each checkpoint, we train an adversary, compute the sensitive task accuracy and evaluate the Pearson correlation between the sensitive task accuracy and the corresponding similarity measure. Results are presented in Fig. 5. Takeaways. Both ADV and MI poorly are correlated with the degree of disentanglement of the learned representations. We find this result not surprising at light of the findings of Xie et al. (2017) and Song and Ermon (2019). All our losses achieve high correlation (   78 ) except for J in the mention task with both encoders, and the FR with BERT on the mention task that achieves medium/low correlation. We believe, that the high correlation showcases the validity of the proposed approaches. Concluding Remarks Malik Boudiaf, Ziko Imtiaz Masud, Jrme Rony, Jose Dolz, Ismail Ben Ayed, and Pablo Piantanida. 2021. Mutual-information based few-shot classification. arXiv preprint arXiv:2106.12252 . Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 . Maximin Coavoux, Shashi Narayan, and Shay B Cohen. 2018. Privacy-preserving neural representations of text. arXiv preprint arXiv:1808.09408 . Pierre Colombo, Emile Chapuis, Matthieu Labeau, and Chlo Clavel. 2021a. Code-switched inspired losses for spoken dialog representations. In Proceedings of the 2021 Conference on Empirical Methods in 2622 Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 83208337. We have introduced a new framework for learning disentangled representations which is faster to train, easier to tune and achieves better results than adversarial or MI-based methods. Our experiments on the fair classification task show that for RNN encoders, our methods relying on the closed-form of similarity measures under a multivariate Gaussian assumption can achieve perfectly disentangled representations with little cost on the main tasks (e.g. using Wasserstein). On BERT representations, our experiments show that the Sinkhorn divergence should be preferred. It can achieves almost perfect disentanglement at little cost but allows for fewer control over the degree of disentanglement.
abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective method objective abstain objective abstain abstain abstain abstain objective method method result abstain abstain abstain other other other other method method method abstain other other other other other other other other other other other other other other other other other other method method method abstain method method method abstain method method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain other abstain method other abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result result abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain