Neural language models are known to have a high capacity for memorization of training samples. This may have serious privacy implications when training models on user content such as email correspondence. Differential privacy (DP), a popular choice to train models with privacy guarantees, comes with significant costs in terms of utility degradation and disparate impact on subgroups of users. In this work, we introduce two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy through (1) the use of a discriminator and (2) the inclusion of a novel triplet-loss term. We compare our methods with DP through extensive evaluation. We show the advantages of our regularizers with favorable utility-privacy trade-off, faster training with the ability to tap into existing optimization approaches, and ensuring uniform treatment of under-represented subgroups. Neural language models (Bengio et al., 2003; Mikolov et al., 2010) have recently seen significant gains in capabilities, and are deployed at scale in several real-world scenarios (Chen et al., 2019; Adam et al., 2020). Training these models on domain-specific user data can further improve their utility. The volume of data required, coupled with the inherent sparsity of natural language which often means all data are unique, opens the door to an array of privacy attacks against models and their training data. Sample memorization poses a substantial risk by enabling model inversion attacks (Carlini et al., 2020; Ramaswamy et al., 2020; Inan et al., 2021). In these attacks, a curious or malevolent user can query a pre-trained language model on any data record with the intention of reconstructing (parts of) training samples 1 . Differential Privacy (DP) (Dwork, 2006) is the gold standard approach to address this issue, thanks to its strong and rigorous privacy guarantees. DP-SGD (Abadi et al., 2016) is a popular method to train neural models with differential privacy guarantees and it works by clipping of the gradients and addition of noise in each update, which provides worst-case guarantees that reflect the likelihood of leaking any attribute of any member of the dataset into the trained model. The worst-case guarantees of differential privacy are not customizable, in other words, they cannot be relaxed to protect only certain attributes. Therefore, DP incurs significant loss to model utility (Tramr and Boneh, 2020). DP training of models is also much slower, with cumbersome hyper-parameter tuning and development (Wu et al., 2017; Subramani et al., 2020). It has also been shown that DP's utility loss is much worse for under-represented groups (Bagdasaryan et al., 2019; Farrand et al., 2020), which can have financial and societal ramifications (Pujol et al., 2020). To address these issues, we relax the strong assumptions of the DP threat model and assume an adversary with finite-capacity (finite statistical, compute, and side information) who attempts to recover sensitive user-level information from the trained model (Carlini et al., 2019). We propose two privacy regularization methods, one based on adversarial training and another on a novel privacy loss term, to jointly optimize for privacy and utility of language models. The main idea of our regularizers is to prevent the last hidden state representation of the language model for an input sequence x from being linked back to the sensitive attribute we are trying to protect, in our case, the identity of the author. We use the last hidden state as it corresponds to the embedding of the sequence x . 2 We 2 Although we consider recurrent neural network-based language models in this work, our approach is applicable in transformer-based language models as well. consider the linkability of the input representation to the sensitive attribute (author) as a proxy since it is commensurate with the linked and linkable information definitions in the General Data Protection Regulation (GDPR Article 29 Working Party, 2014). By framing privacy as an optimization problem, we can apply the well-developed machinery of large-scale gradient-based optimization, enabling us to train models at scale while jointly tuning for an optimal privacy-utility trade-off. To validate our approach, we develop an evaluation framework for assessing a model's privacy loss. We employ the exposure metric introduced in (Carlini et al., 2019) and introduce a reconstruction (tab) attack as a realistic scenario to evaluate and compare LSTM language models trained using our regularization with those trained with differential privacy, on Avocado (Oard et al., 2015) and Reddit (Vlske et al., 2017) datasets. We also empirically demonstrate that, unlike DP, our technique does not have disparate impacts on underrepresented groups. Our work is closely related to (Coavoux et al., 2018) and (Li et al., 2018). Coavoux et al. consider an attacker who eavesdrops on the hidden representations of a pre-trained model during inference and tries to recover information about the input text. Adversarial training is used as a mitigation to reduce the attacker's performance (Wang et al., 2019). Li et al. use adversarial training to protect private author attributes such as age or gender, in learned text representations for part-of-speech tagging and sentiment analysis to gain better performance on out-of-domain corpora. We, on the other hand, use adversarial training and a triplet-based regularization to train private language models that do not memorize sensitive user information, which has not been explored before. We evaluate our models accordingly, by trying to extract training samples. Prior work has studied membership inference attacks against models (Shokri et al., 2017; Yeom can consider the representation corresponding to the special token [CLS] as the embedding of the sequence x . et al., 2018; Song and Shmatikov, 2019), however, our regularizations do not target these attacks. Figure 1 shows our first proposed regularizer which is adversarial in nature. We feed an input text sequence x to the language model and extract the last hidden state representation of the model for x ; denoted by h x . h x is then fed to a discriminator parameterized by  d , which plays the role of an attacker who attempts to predict what the sensitive label (in our case, the author, y ) for x is. The output probability distribution of the discriminator for the input h x , p d = Pr( | h x ;  d ) is then used to compute both the privacy loss LLM-P of the language model and the discriminator loss LD-CE . During training, the discriminator optimizes for better linking of the last hidden state representations to the authors. Thus, the discriminator loss is LD-CE ( h x , y ;  d ) =  log Pr( y | h x ;  d ) . Conversely, the language model optimizes  lm such that it (1) improves the utility of the language model and (2) flattens the probability distribution over authors. Thus, we devise the following loss function: LLM ( x ;  d ,  lm )= LLM-CE +  LLM-P (1) LLM-CE is the utility loss, for which we use conventional cross entropy loss over the next-word predictions. LLM-P is the privacy loss : LLM-P ( h x ;  d ) =  1 MM (cid:88) c =1 log Pr( c | h x ;  d ) (2) i.e. the KL divergence between the distribution over authors and the uniform distribution where M is the number of classes (authors). The goal of this term is to drive the discriminator to predict randomly uniform outputs (Raval et al., 2019). The reason we devised this loss as opposed to using 0 50 100 150 200 0 10 20 30 40 50 E x p o s u r e Canary Repetition Unmitigated DP Triplet Adversarial L D-CE is that we do not just want the discriminator to assign zero probability to the correct author, we want p d to be uniform so that it has no information about the correct author. Hyperparameter  allows for trading off privacy and utility. One potential downside of the proposed adversarial regularizer is that the capacity of the discriminator must scale with the number of authors, and thus the size of the training data. To better accommodate the larger number of authors in large datasets, we investigate another regularizer that does not require a discriminator. We build on the intuition that to obfuscate an attribute, we can increase the distance between representations of samples that have the same label for that attribute while decreasing the distance between samples with different labels. To this end, we use the language model loss ( LLM ) of the previous section (Eq 1), and we set the privacy loss to be the triplet loss: LLM-P = (cid:107) h x  h p (cid:107) 2  (cid:107) h x  h n (cid:107) 2 (3) The triplet loss is commonly used in vision tasks for training embeddings that map images from the same category to neighboring points in the embedding space (Chechik et al., 2010). We, however, invert this loss and use it for an opposite purpose: privacy regularization. During the training of the language model, we select a baseline sample, x , a positive sample p (with different sensitive label) and a negative sample n (with the same sensitive label) and feed them through the language model and extract the last hidden states h x , h p and h n , respectively. We find the l 2 distance between h x , h p , and h n and based on their labels, add them to or subtract them from the loss. To implement this, in practice, we sample a baseline batch and a second auxiliary batch during training. We feed both the baseline batch ( x ) and the auxiliary batch ( a ) through the language model and extract the last hidden states. We then calculate the distance between the last hidden states of the corresponding samples in the two batches. If the samples have different labels for the sensitive attribute (author), we add their distance to the loss, otherwise, we subtract it. The privacy loss becomes: LLM-P = (cid:88) i : y xi = y ai (cid:107) h x i  h a i (cid:107) 2  (cid:88) j : y xj (cid:54) = y aj (cid:107) h x j  h a j (cid:107) 2 (4) 3 Evaluation In our experiments, we use a subset of the Avocado corporate email dataset (Oard et al., 2015) with 100 users and 60,000 samples and a subset of Reddit dataset (Vlske et al., 2017) with 10,000 users and 3 million samples. Both of these datasets are in English, covering formal and informal writing. We create a 80 / 20% training/test set split. We use a two-layer LSTM model as the language model for the next-word prediction task. We compare models trained with our proposed regularizer to differentially private (DP) ones (Abadi et al., 2016). For the privacy accounting, we use Gaussian differential privacy (Bu et al., 2019). We use language model perplexity as a measure of utility. Due to space limitations, we focus evaluations on privacy metrics for several set levels of achieved test perplexity, listed in Table 1 in the appendix. See appendix A.2 for a more detailed description of the experimental setup and extra analysis of overheads and complexity of each regularizer. Privacy measurements w/ exposure metric. To empirically compare the privacy of our methods to that of DP, we adopt the exposure metric introduced in (Carlini et al., 2019). The higher the exposure of a sequence, the more the model's memorization and the easier it is to extract the sequence from the language model. To measure exposure we insert sequences of five random words (canaries) to the training data (appendix A.5). We insert unique 0% 1% 2% 3% 4% 5% Unmitigated DP Triplet Adversarial A tt a c k A cc u r a c y Synthesized Canary Real Canary Figure 2 shows the exposure results per canary repetition. These results are averaged over all the users. In each sub-figure, the perplexities of the models are similar, hence we can compare the privacy levels at similar utilities. Fig. 2a compares trained models using different techniques on the Avocado dataset, where they all have relatively high perplexities compared to a fully trained conventional model (Table 1). Fig. 2b has the same setup, however, the models have lower perplexities. Naturally, for having better utility we are trading off privacy, which can be seen by comparing the exposure values in these two figures and observing that the second one has higher exposure values (lower privacy). Finally, Fig. 2c shows the exposure results for Reddit. In all cases, we see that the unmitigated model has the highest exposure, as expected. We also observe that for canaries (patterns) that are repeated more than 9 times (for each user), our mitigation offers lower exposure compared to DP, especially in the high perplexity case. This is because clipping and noise addition in DP is attribute and data-agnostic, meaning that noise is added to all samples regardless of whether or not they contain sensitive information. Therefore, repeated patterns are less protected. If we want to protect a pattern with n repetitions, we would need to apply noise that is n  larger, which would degrade the utility gravely and would not yield the same perplexity. For lower repetition canaries, our mitigations have comparable performance to DP. For all these experiments the Gaussian differential privacy criterion  is extremely large ( 10 20 ), which practically yields (cid:15)   . We also experimented with lower (cid:15) values (e.g. (cid:15)  7 ), however, it yields a model with perplexity of 650, having an extremely low utility. and see if the entire sequence is reconstructed using the language model. We report the rate of correct reconstruction of canaries as the accuracy of the attack. We use the synthetic canaries from the previous experiment, and also select real canaries from the training corpus to create a real-world scenario. Fig. 3a shows that for a high perplexity model, the accuracy of the tab attack on synthesized canaries is very small, even for the unmitigated model. The unmitigated model reaches the designated perplexity in less than an epoch, and hence it does not memorize the canaries. For the real canaries, however, the memorization is higher, since they follow grammatical rules. In the lower perplexity case of Fig. 3b, we see that the synthesized canaries are mostly memorized by the unmitigated model. Our mitigations outperform DP, especially for the synthesized canaries. DP is not context-sensitive and applies the same amount of noise to all samples, thereby leaving correlated and higher repeated samples less-protected. Our mitigations, however, learn what sequences are link-able to their authors, and obfuscate them such that they no longer leak the identifying secret. Effect on under-represented users. Differential privacy has disparate impact on the accuracy of different subgroups of the dataset (Bagdasaryan et al., 2019). Here, we want to measure the effect of our mitigations on the utility of the model among users with various data samples. For each user, we measure the average perplexity of the model for their samples in the test set, and then subtract this from the same value for an unmitigated model. This would yield the average drop in utility, per user. We compare the utility drop of well-represented users to under-represented ones by taking the top 5 users with the most samples and the bottom 5 users with the fewest samples from Avocado dataset. We then measure the average utility drop over each group of 5 users on the test set. Figure 3c shows these results. We see that differential privacy has disparate impact, 29 points, on the two sub-groups of users (authors), whereas this gap is only 7 points for models trained with our mitigations. It's important to remember that in general, distinguishing under-represented users from those whose data is similar to others but who have contributed fewer samples is a difficult task. However, for Figure 3c's results, if these users' data came from the same distribution as the ones with lots of samples (i.e. if these people were merely less-contributing), the utility loss would be similar for all groups when applying user-level DP (what we use). DP's disparate impact on the utility loss for these two groups suggests that, in our case, the less-contributing authors are probably also underrepresented. This work introduces two privacy mitigation methods to jointly optimize for privacy and utility. Extensive experiments show that our approach provides comparable and in certain cases a higher level of privacy compared to differentially private model training. We further empirically demonstrate, that our methods do not exhibit disparate impacts on under-represented groups and have significantly less overhead on training performance. The Avocado corpus is licensed for research applications under strict terms intended to protect the privacy of the correspondents. While the end-user license agreement does not indicate what consent was granted by the participants, one term of the license is that End user will obtain whatever training and approval is required by their organization for working with human subjects data, which we have obtained (more details in (Oard et al., 2015)). While handling sensitive email data (Avocado) we made sure to abide by the terms of its end-user license agreement (EULA) which has provisions to protect the privacy of members of the corpus. Furthermore, we took measures such as scrubbing named entities before using the data for model training. The over-arching goal of our work is to contribute to language model development that protects the privacy rights of users who contribute their data. While we rigorously evaluated our models by applying state-of-the-art attacks, deploying these models in real-world setups requires further verification that users' privacy is preserved. The authors would like to thank the anonymous reviewers and meta-reviewers for their helpful feedback. We also thank Peter Kairouz and Mohammadkazem Taram for insightful discussions. Additionally, we thank our MSR colleagues and UCSD Berg Lab for their helpful comments and feedback.
abstain abstain abstain objective method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain method method abstain objective objective method objective abstain abstain abstain abstain objective method method method method method abstain abstain abstain abstain method abstain abstain method method abstain abstain method method method method method method method method method method method abstain method method method method method method abstain abstain method abstain method method abstain abstain method abstain abstain method abstain method method abstain abstain method method abstain method abstain method method abstain abstain abstain method method abstain method abstain abstain method method abstain method method abstain method abstain method method method result objective abstain abstain abstain method abstain method other other other