Exploiting natural language processing in the clinical domain requires de-identification, i.e., anonymization of personal information in texts. However, current research considers de-identification and downstream tasks, such as concept extraction, only in isolation and does not study the effects of de-identification on other tasks. In this paper, we close this gap by reporting concept extraction performance on automatically anonymized data and investigating joint models for de-identification and concept extraction. In particular, we propose a stacked model with restricted access to privacy-sensitive information and a multitask model. We set the new state of the art on benchmark datasets in English (96.1% F1 for de-identification and 88.9% F1 for concept extraction) and Spanish (91.4% F1 for concept extraction). In the clinical or biomedical domain, natural language processing (NLP) could significantly improve the efficiency and effectiveness of processes. For example, the extraction of structured information from clinical narratives can help in decision making or drug repurposing (Mari-mon et al., 2019). However, the automatic processing of documents with privacy-sensitive content is restricted due to the necessity of applying anonymization techniques. Text anonymization, also called de-identification, aims at detecting and replacing protected health information (PHI), 1 such as patient names, age and phone numbers, as shown in the upper part of Figure 1. Recent studies show that automatic de-identification leads to 1 PHI types are typically defined by governments, for instance in the Health Insurance Portability and Accountability Act (HIPAA) of the United States. AGEPROBLEM Patrick died of myocardial infarction (MI) at 76. PHI terms  concept  Figure 1: Sentence with annotations of the two tasks. promising results (Stubbs and Uzuner, 2015; Marimon et al., 2019). A severe limitation of current approaches in research, however, is that de-identification is typically addressed in isolation but not together with a downstream task, such as concept extraction (CE) from medical texts (Gonzalez-Agirre et al., 2019; Uzuner et al., 2011). Instead, the downstream task models are trained and evaluated on the non-anonymized data, and it remains unclear how de-identification affects their performance in real-world settings. In this paper, we argue that to evaluate the effectiveness of NLP in the medical domain, the tasks of de-identification and information extraction should be analyzed together. Our contributions are as follows: We close this gap and analyze the effect of de-identification on clinical concept extraction. Moreover, we consider the two tasks jointly and propose two end-to-end models: A multitask model that shares the input representation across tasks, and a stacked model that trains a pipeline of de-identification and concept extraction in an end-to-end manner. For the stacked model, we propose to use a masked embedding layer to restrict the access of the concept detector to privacy-sensitive information and train it on an anonymized version of the data. To make the model differentiable, we use the Gumbel softmax trick (Maddison et al., 2017; Jang et al., 2017). We conduct experiments on clinical benchmark datasets in English and Spanish. Our results indicate that de-identification does not affect CE models negatively, but has even a slight positive effect on the results, probably because de-identification homogenizes the input for CE. Modeling both tasks jointly leads to better results than treating de-identification as a pure preprocessing step, resulting in new state-of-the-art performance for CE. For future research, we publish our code. 2 2 Related Work While many works propose joint training for other NLP tasks (i.a., Finkel and Manning, 2009; Miwa and Sasaki, 2014), including multitask learning (i.a., Collobert and Weston, 2008; Klerke et al., 2016; Sgaard and Goldberg, 2016) and stacking of pipeline components (i.a., Miwa and Bansal, 2016), we are to the best of our knowledge the first to combine de-identification with an information extraction task. In this section, we report related work in those two fields. The increasing importance of de-identification is reflected in the number of shared tasks (Uzuner et al., 2007; Stubbs and Uzuner, 2015; Marimon et al., 2019). State-of-the-art methods for de-identification typically rely on recurrent neural networks (RNNs) (Dernoncourt et al., 2016; Lange et al., 2019b; Kajiyama et al., 2018). Feutry et al. (2018) and Friedrich et al. (2019) create pseudo-de-identified text representations with adversarial training. In particular, they replace personal information, such as names, by other names. Zhao et al. (2018) augment the training data by creating more general text skeletons, e.g., by replacing rare words, such as names, by a 2 https://github.com/boschresearch/ joint_anonymization_extraction special unknown token. Compared to these works, we exploit the advantages of both approaches and replace personal information by their class names as placeholders. This approach is not only common for de-identification (Johnson et al., 2016), but also for relation extraction where entities are often either replaced by their type or enriched with type information (i.a., Zhang et al., 2017; Miwa and Sasaki, 2014). We further motivate our choice in Section 3.2. Another difference to the above mentioned works is that we do not augment the training data for our de-identification model. Analogously, there have been a series of shared tasks for information extraction in the clinical and biomedical domain (Uzuner et al., 2011; Sun et al., 2013; Krallinger et al., 2015; Gonzalez-Agirre et al., 2019). Models for these tasks often either rely on hand-crafted features (Leaman et al., 2015; Xu et al., 2012) or RNNs (Hemati and Mehler, 2019; Korvigo et al., 2018; Tourille et al., 2018). Newman-Griffis and Zirikly (2018) study the performance of RNNs for medical named entity recognition in the context of patient mobility and find that they benefit from domain adaption. In contrast to previous work, we investigate the usage of de-identified texts as input for clinical concept extraction models and propose to jointly model de-identification and concept extraction. In this section, we present our systems for the two individual tasks and our proposed joint models. Figure 2 shows the respective architectures. We model both document anonymization (ANON) and clinical concept extraction (CE) as sequence labeling problems and apply a bidirectional long short-term memory (BiLSTM) network (Hochre-iter and Schmidhuber, 1997) with a conditional random field (CRF) output layer (Lafferty et al., 2001), similar to Lample et al. (2016). In recent works on clinical de-identification and CE, this architecture was shown to be very promising (Mari-mon et al., 2019; Gonzalez-Agirre et al., 2019). Each token is represented with a concatenation of different pre-trained language-specific embeddings: byte-pair-encoding (Heinzerling and Strube, 2018), fastText (Bojanowski et al., 2017) and FLAIR (Akbik et al., 2018). For Spanish, we also include multilingual BERT embeddings (De-vlin et al., 2019). Further, we include the following domain-specific embeddings: clinical BERT for English pre-trained on discharge summaries (Alsentzer et al., 2019) and clinical fastText for Spanish pre-trained on the Spanish E-health corpus from the Scielo archive (Soares et al., 2019). To assess the effects of de-identification on CE, we first apply the de-identification model to anonymize the CE dataset and then evaluate the CE model on the anonymized data. We refer to this approach as PIPELINE model (see Figure 2a). For anonymization, we replace each detected privacy-sensitive term with a placeholder of its PHI type, i.e., there is one placeholder per type. This replacement choice has advantages over the alternatives described in Section 2. Compared to replacing personal information with alternative names, it leads to a more general text and thus, homogenizes the input for the downstream-task classifier. Compared to replacing all personal information with the same token, the resulting text is more specific, allowing the downstream-task classifier to take into account which kind of personal information was mentioned. Thus, the approach is a trade-off between more homogeneous input and more fine-grained information for the downstream-task classifier. Instead of using a sequential pipeline, we propose to train both tasks jointly. For this, we test two approaches: a multitask model and a stacked model. In the MULTITASK model (Figure 2b), the weights up to the BiLSTM layer are shared across both tasks. For each task, we add a task-specific hidden layer with ReLU activation and a CRF output layer. Note that in this architecture, the CE model has access to the original, privacy-sensitive data. We also propose a STACKED model (Figure 2c), where only the de-identification part has access to the privacy-sensitive information. The access of the CE part is restricted by a masked embedding layer as described in the following. Masked Embedding Layer. The masked embedding layer ensures that the CE model does not have access to privacy-sensitive information by replacing the input embeddings of privacy-sensitive tokens by PHI-class embeddings which are randomly initialized and fine-tuned during training. This is depicted in Figure 3. Gumbel Softmax Trick. The masked embedding layer requires a discrete output from the de-identification part. In order to ensure that the model stays fully differentiable, we use the Gumbel softmax trick (Maddison et al., 2017; Jang et al., 2017). It approximates categorical samples with a continuous distribution on the simplex and computes gradients for backpropagation with the reparameterization trick. The Gumbel softmax function has the following form: y k = exp((log  k + G k ) / ) (cid:80) Ki =1 exp((log  i + G i ) / ) (1) with  1 , ... K being the unnormalized output scores from the de-identification layer, G 1 , ..., GK being i.i.d samples drawn from Gumbel(0, 1) and  being a temperature. For   0 , the distribution becomes identical to the categorical distribution. The masked embedding layer takes the output of the Gumbel softmax (i.e., an anonymization label) and if the label is a PHI class and requires anonymization, the masked embedding layer uses English (i2b2) Spanish ANON CE ANON CE # classes 24 3 22 3 train (# tokens) 45,793 16,315 15,903 8,068 dev (# tokens) 5,088 -8,277 3,748 test (# tokens) 32,587 27,625 7,966 3,930 Table 1: Dataset statistics. the respective PHI class embedding vector, otherwise it uses the original embedding vector. In this section, we describe the datasets used in our experiments, and training details for our models. Finally, we present our results and analysis. We evaluate our models on corpora from the clinical domain in English and Spanish. For English, we use the data from the i2b2 2010 CE task (Uzuner et al., 2011) and the i2b2 2014 de-identification task (Stubbs and Uzuner, 2015). For Spanish, we use the MEDDOCAN (Marimon et al., 2019) corpus for de-identification and the PharmaCoNER corpus (Gonzalez-Agirre et al., 2019) for CE. As PharmaCoNER is a subset of MEDDOCAN, we have both gold-standard concept and de-identification annotations for this data. Data Preprocessing. We used the preprocessing scripts from Alsentzer et al. (2019) for the English i2b2 corpora and the Spanish Clinical Case Corpus tokenizer (Intxaurrondo, 2019) for both Spanish corpora. We noticed that the Spanish tokenizer sometimes merges multi-word expressions into a single token joined with underscores for contiguous words. As a result, some tokens cannot be aligned with the corresponding entity annotations. To address this, we split those tokens into their components in a postprocessing step. Table 1 shows statistics about corpora sizes. Hyperparameters. The embeddings have 300 (byte-pair-encoding), 300 (fastText) and 4,048 (FLAIR) dimensions. For English, we use clinical BERT embeddings with 768 dimensions which are constructed by averaging the last four layers with the scalar mix operation proposed by Liu et al. (2019). We concatenate all embeddings to one input vector, resulting in a total input dimensionality of 5,416. Analogously, we use multilingual BERT (768 dim.) and domain-specific fastText embed-Models English Spanish Yang and Garibaldi (2015) 96.0 Zhao et al. (2018) 94.0 Alsentzer et al. (2019) 93.0 Lange et al. (2019b) 97.0 Hassan et al. (2019) 96.3 Perez et al. (2019) 96.0 Our PIPELINE (ANON only) 96.1 96.8 Our STACKED 95.9 96.8 Our MULTITASK 95.2  96.7 Table 2: F1 results for de-identification. dings (100 dim.) for Spanish, resulting in 5,516 input dimensions. For the LSTM, we use 256 hidden units per direction. The task-specific hidden layer of the multitask model has 128 units. Training. For training, we use stochastic gradient descent with a learning rate of 0.1 and a batch size of 32 sentences. The learning rate is halved after 3 consecutive epochs without improvements on the development set. For the joint models, we pretrain the anonymization part for 3 epochs and, then, use a higher initial learning rate of 0.2 for the concept extraction part. We perform early stopping on the development set. If no development set was provided by the corpus (i2b2 2010 corpus), we held out 10% of the training set as development set. Note that we use the same hyperparameters for all our models and all tasks, which were tuned on the Spanish concept extraction data. Evaluation. We train each model with three random initializations and report F1 for exact matching for the best model in all experiments. We perform statistical significance testing to check if our joint models are better than the PIPELINE model. We use paired permutation testing with 2 20 permutations and a significance level of 0.05. Table 2 shows that the de-identification component of our PIPELINE model which was trained on the single task of de-identification sets the new state of the art on English and performs comparable on Spanish. The performance difference to our prior work (Lange et al., 2019b) is due to a slightly different set of input embeddings. However, we found no statistically significant differences to that model. The de-identification performance of STACKED is comparable to the PIPELINE model, the de-identification performance of MULTITASK Models No PHI English Spanish de Bruijn et al. (2010) no 85.2 Xu et al. (2012) no 84.9 Alsentzer et al. (2019) no 87.7 Sun and Yang (2019) no 89.2 Lange et al. (2019a) no 88.6 Our PIPELINE yes 88.0 89.6 Our STACKED yes 88.7  90.0 Our MULTITASK no 88.9  90.3  Xiong et al. (2019)  no 91.1 Stoeckel et al. (2019)  no 90.5 Our MULTITASK  no 91.4  Table 3: F1 results for concept extraction. is slightly lower, however, we only found statistically significant differences for the MULTITASK model for English. The results for our concept extraction models in comparison to state of the art are shown in Table 3. We set the new state of the art on both languages. While in the PIPELINE setting, the CE performance is slightly lower (as it has been trained on the non-anonymized texts but is evaluated on the de-identification output), training de-identification and CE jointly leads to considerable improvements for both STACKED and MULTITASK with statistically significant differences for both models in English and for MULTITASK also in Spanish. Especially the results of STACKED in comparison to PIPELINE shows that end-to-end training of the two steps is promising, while still preserving privacy aspects during model training by restricting internal access to PHI tokens. The performance of each embedding used in our experiments is shown in Table 4. As mentioned before, we did not include multilingual BERT embeddings for English, but show their results for completeness. Finally, we analyze the impact of de-identification on CE. The results for training and testing our CE model on different inputs (non-anonymized vs. anonymized) are shown in Table 5. We restrict our analysis to Spanish since the data is labeled with both de-identification and concept information (see Section 4.1). Thus, we can also investigate the difference between gold and predicted de-identification labels. The CE model benefits from being trained and evaluated on anonymized data (lines 4-6). However, it hurts to train on non-anonymized data and evaluate on predicted de-identification labels (line 1 vs. 2) and vice versa (line 1 vs. 3). This supports our motivation that it is necessary to investigate anonymization and downstream applications together. The difference of training on gold vs. predicted de-identification labels (lines 4-6) is only marginal, suggesting that state-of-the-art de-identification systems are good enough to be used in such settings. In this paper, we close the gap and consider de-identification of clinical text together with concept extraction, a possible downstream application. We investigate the effects of de-identification on concept extraction and show that it positively influ-ences the concept extraction performance. We propose two models to learn both tasks jointly, a multitask model and a stacked model, and set the new state of the art on medical concept extraction benchmark datasets for English and Spanish. We would like to thank the members of the BCAI NLP&KRR research group and the anonymous reviewers for their helpful comments.
abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result result abstain abstain abstain result result abstain result result abstain abstain abstain abstain abstain method objective objective other