Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s). We explore the implications of this phenomenon for model fairness across demographic groups in clinical prediction tasks over electronic health records (EHR) in MIMIC-III  the standard dataset in clinical NLP research. Apparent subgroup performance varies substantially for seeds that yield similar overall performance, although there is no evidence of a trade-off between overall and subgroup performance. However, we also find that the small sample sizes inherent to looking at intersections of minority groups and somewhat rare conditions limit our ability to accurately estimate disparities. Further, we find that jointly optimizing for high overall performance and low disparities does not yield statistically significant improvements. Our results suggest that fairness work using MIMIC-III should carefully account for variations in apparent differences that may arise from stochas-ticity and small sample sizes. Fine-tuning pre-trained transformers (Vaswani et al., 2017) such as BERT (Devlin et al., 2019) has become the dominant paradigm in NLP, owing to their performance across a range of downstream tasks. Clinical NLP  in which we often aim to make predictions on the basis of notes in electronic health records (EHRs)  is no exception (Alsentzer et al., 2019). However, fine-tuning large networks is a stochastic process. Performance can vary considerably as a function of hyperparame-ter choice, and many parameter sets can yield the same validation accuracy (i.e., the model is not identifiable), and more generally the problem is underspecified (D'Amour et al., 2020). Recent work has demonstrated that the choice of random seeds alone can have dramatic impact on model performance in NLP and beyond, even when all men women white black asian hispanic group  0 . other hyper-parameters are kept fixed (Phang et al., 2018; Dodge et al., 2020; D'Amour et al., 2020). In this work, we explore the intersection of randomness and fairness in the context of clinical NLP. Fairness is a particularly acute concern in clinical predictive tasks, given the potential of such models to influence treatment decisions. This has motivated work investigating biases in predictive models trained over EHR (Zhang et al., 2020; Chen et al., 2018, 2019, 2020a; Pfohl et al., 2020; Chen et al., 2020b; Tripathi et al., 2020). We investigate the impact of random seeds on the fairness of fine-tuned classifiers with respect to demographic characteristics such as gender and ethnicity. There are many definitions of algorithmic fairness which formalize different desired properties (Mehrabi et al., 2019). Following prior work, here we adopt a simple measure: The mean differences in model performance across demographic subgroups (Chen et al., 2019). We find that, on the popular MIMIC-III dataset (Johnson et al., 2016), seeds with comparable validation performance can give rise to large variations in disparities across demographic subgroups (Figure 1). We investigate the variability of overall model performance and fairness across random seeds for a set of clinical prediction tasks derived from the Mul-tiparameter Intelligence Monitoring in Intensive Care (MIMIC-III) set of Electronic Health Records (EHRs; Johnson et al. 2016). For each task, we train a classifier on top of the contextualized representations of a BERT (Devlin et al., 2019) model pretrained over EHR data (Alsentzer et al., 2019). Following recent work exploring randomness and fine-tuning, we consider the seeds used to shuffle the training data and to initialize the model parameters independently (Dodge et al., 2020). Specifically, we generate K = 1000 pairs of shuffling and initialization seeds by sampling from a uniform distribution U (0 , 10000) . For each seed pair, we measure the overall performance as well as the performance for each demographic subgroup in terms of the Area Under the ROC Curve (AUC). MIMIC-III is a database of deidentified EHR comprising over 40k patients admitted to the intensive care unit of the Beth Israel Deaconess Medical Center between 2001 and 2012 (Johnson et al., 2016). It comprises structured variables including vital sign measurements, lab test results, and medications. It also contains clinical notes (e.g., doctor and nursing notes, radiology reports, and discharge summaries), which are the focus of our analysis. MIMIC-III contains demographic information, including potentially sensitive attributes such as ethnicity/race, sex, spoken language, religion, and insurance status (which may be seen as a proxy for socioeconomic status (Chen et al., 2019)). We are interested in the interaction between randomness and fairness in clinical predictions. Following recent prior work (Zhang et al., 2020) on the latter, we focus our analyses on two benchmark tasks proposed by Harutyunyan et al. (2019): In-hospital Mortality (IHM) Predict risk of in-hospital mortality based on the first 48 hours of an ICU stay. Phenotype Classification (PC) Classify which of 25 acute or chronic conditions (e.g., acute cerebrovascular disease, chronic kidney disease) are present in a given patient ICU stay record. Similar to Zhang et al. (2020), we treat each condition as an independent binary classification task. Table A.1 in the Appendix enumerates the full set of conditions and their respective prevalences. We extracted training and test datasets for these tasks using the same pre-processing pipeline as Zhang et al. (2020). 1 We kept the same data splits and reserved 20% of the training data as validation set per task. For each patient, we collected their clinical notes, as well as their gender and race/ethnicity (as recorded in the EHR). The notes were filtered according to the categories Nurse , Physician and Nursing/Other to avoid notes of poor semantic quality, as suggested by Zhang et al. (2020). Patients without relevant clinical notes were discarded, resulting in 11384 / 2591 and 22033 / 4919 training/test examples for the IHM and PC tasks, respectively. It should be noted that these datasets are highly imbalanced both in terms of labels and demographic distribution with 55% Male , 85% White , 9% Black , 3% Asian and 3% Hispanic patients. Table 1 shows the distribution of sample sizes across subgroups for each benchmark. We define text classifiers for clinical tasks that map clinical notes corresponding to individual patients to binary labels. We extract contextualized embed-dings from notes using a pretrained Transformer encoder and then map these to outputs (predictions) via a linear layer. Transformers are feedforward networks and require fixed-length inputs. To handle longer sequences, we adopt an approach used in prior works (Huang et al., 2019; Zhang et al., 2020). Given an input sequence, we: (1) Extract N subsequences with sizes equal to that expected by the Transformer input layer; (2) Make individual 1 https://github.com/MLforHealth/ HurtfulWords predictions on the basis of each subsequence, and; (3) Then aggregate them into a final prediction. More formally, an encoder  operates over inputs of size E with H -dimensional hidden layers. Given a patient's clinical notes X , we extract a set of N subsequences of length E , x = {{ w 11 , . . . , w 1 E } , . . . , { w N 1 , . . . , w NE }}  X We construct a matrix Z  RH  N such that the n th column represents subsequence x n , Z [: , n ] =  ( x n ) = (cid:80) j z x n j where z x n j  RH is the embedding produced by the last hidden layer of the encoder for token j in the context of x n . We then use a linear layer followed by a sigmoid activation to produce a prediction vector Y , encoding the class conditional probabilities for each subsequence. This vector is then used to calculate the final probability as P ( Y = 1 | Y ) = Y max + Y mean N/c 1 + N/c , (1) where c is a scaling factor, which we set to c = 2 , following Huang et al. (2019). We implement classifiers with PyTorch using the Transformer encoders from the huggingface 2 library (Wolf et al., 2019). We initialize models to weights from ClinicalBERT (Huang et al., 2019), which was trained over scientific literature and clinical notes. We train classifiers on the most recent N = 10 subsequences of E = 512 tokens from the notes associated with each patient. We train using the ADAM optimizer (Kingma and Ba, 2014) for 500 epochs with early stopping. We set the learning rate to  = 0 . 01 , which we found to have the best validation performance on average across all tasks. We compare the overall performance with the performance for each subgroup as a function of random seeds. Figure 2 shows the overall performance (left) along with the gap between the best and worst observed subgroup AUCs (right), across tasks. We observe a large variance in both the overall performance and the gap. The former observation corroborates previous findings (Dodge et al., 2020). To quantify how random seeds affect individual subgroups, we measure the the absolute differences (  s) between overall and subgroup performances. We then evaluate whether there are correlations between overall performance and subgroup  s. Figures 3 and 4 present the results for the Shock phenotype classification task  one of the tasks with largest disparities observed in prior work (Zhang et al., 2020). Similar trends were found for the remaining tasks, and we report all results in the Appendix (Figures A.2-A.3 and A.4-A.6). Figure 3 shows that the performance of all subgroups varies significantly across random seeds and that variances are higher for minority groups. Larger variations in minority subgroups are to be expected, as any empirical estimate will have a variance that is inversely proportional to the sample size of a group. In Figure 4, we observe that there seem to be two distinct clusters of seeds: One corresponding to high performing models (right of plots), and another to suboptimal models. 3 While the best performing models tend to have a lower variance of subgroup performance, there is otherwise no clear relationship between overall and subgroup performance. Indeed, we find that many models with similar overall performance correspond to widely different subgroup  s, particularly for the minority groups. To explore the implications of this phenomenon, we simulate a grid search over all the random seeds on the validation set. We select the best seed along with all other seeds with similar performance (i.e., within a difference of (cid:15) = 0 . 01 absolute AUC). Figure 1 shows the test set subgroup performance  s, for the best validation seeds, in the Shock phenotype classification task (see Figures A.7-A.8 for the other tasks). Figure 5 summarizes the overall performance (left) along with the subgroup performance gap (right) across tasks. We can see that selecting seeds on the basis of overall performance helps to reduce the subgroup performance gap (compare the right subplots in Figures 2 and 5). However, the top performing models show disparities with respect to both gender and ethnicity, suggesting that these models maximize performance for some groups at the expense of others. Moreover, we find multiple seeds with similar levels of validation performance that correspond to very different subgroup  s . Since we have not encoded any model selection preferences into the pipeline this variance may re-flect a form of underspecification . Can we define 3 Dodge et al. (2020) also found that some seeds performed consistently well across all the evaluated tasks, while others always performed poorly. criteria that explicitly accounts for subgroup performance? We could then ask whether it is possible to maximize both fairness and overall performance with respect to random seeds. We repeated the grid search experiments with simple criteria that incorporate some notion of subgroup performance, such as selecting the seeds that: (a) maximize subgroup macro-average performance; (b) minimize the average subgroup  ; and (c) maximize the overall performance minus the average subgroup  . To account for the effect the sample sizes on the apparent subgroup performance, we directly compare subgroup  s for each random seed on the validation set and the test set. We find that correlations between validation set and test set fairness are either non-existent or very weak in most tasks. may produce models with similar validation performance but very different levels of apparent fair-ness' as a result of varying the random seed alone. However, the fact that training-set and validation-set fairness are not reliable indicators of test-set fairness suggests that variance due to small subset sizes may be significant. This is in some sense not surprising, given the combination of pronounced class imbalance and small subgroup samples in this data (see Tables 1 and A.1). To confirm this, we repeat the experiments on a subset of the test data containing the same number of examples (equal to the smallest subgroup) for all groups, including majority groups. Evaluating all subgroups using small samples yields similarly high variances in performance  (Figure 6 and Appendix Figure A.1), which con-0 . firms that the sample size is a significant factor in the variation of apparent model performance across random seeds. These findings suggest that work investigating the fairness of fine-tuned classifiers should be careful to account for: a) model variability due the choice of random seeds; and b) variance in performance estimates due to small sample sizes. See Appendix Section B for an illustrative example. These observations are relevant for research using MIMIC-III, and for any corpora with similar properties, namely the combination of class imbalance and comparatively small subgroup sizes, which is likely to be present in EHR data where conditions are relatively rare and one is interested in fairness to minority groups (which are smaller by definition). We have investigated the impact of random seeds on the fairness of fine-tuned pre-trained models for clinical tasks. Specifically, we measured gaps in performance across gender and racial subgroups as a function of the choice of random seeds for data shuffling and parameter initialization. In line with prior work, we found that classifiers trained on MIMIC-III data are often biased with respect to demographic subgroups. The contribution of this work is the empirical confirmation that choice of random seed alone significantly affects the apparent bias: Seeds that yield comparable performance in aggregate on the validation data correspond to very different performances on subgroups in test data. Our analyses corroborate Dodge et al. (2020)'s findings on the importance of carefully chosen random seeds, but also suggest that an equal amount of attention should be payed to the impact of these choices on model fairness. However, interpretation of these results is complicated by sample size effects. While MIMIC-III is in itself a large dataset, it also exhibits significant imbalance, both in terms of subgroups of patients and the prevalence of medical conditions. These imbalances compound when considering subsets of patients in the context of specific prediction tasks, which often leads to small sample sizes for minority subgroups. While we observed higher apparent variances for demographic minorities, our results also suggest that these variances can in large part be explained by the smaller sample sizes. Indeed, we found the variances in subgroup performance to be inversely proportional to the size of the subgroup. Fairness has rightly been an issue of increasing concern within the NLP community. This issue is particularly important in clinical NLP, given the potential that such models may ultimately have on patient health. We have investigated the degree to which different subgroup performances may be observed even fixing the (aggregate) validation data performance; we find wide variances across subgroups. That said, this work also highlights inherent limitations of using MIMIC-III (the standard dataset for clinical NLP) to evaluate the fairness of models, given the relatively small samples of patients that belong to demographic groups of interest. We hope these contributions encourage continued research into fairness in the context of clinical NLP. We would like to thank Darius Irani for his contribution in replicating the experiments from (Zhang et al., 2020). This material is based upon work supported in part by the National Science Foundation under Grant No. 1901117.
abstain objective abstain result result result abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain method result method method method method method abstain abstain method abstain method method abstain method abstain method method method abstain abstain abstain abstain method method abstain method other abstain method method method method method method method method method abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method result objective method abstain abstain abstain result result abstain abstain abstain abstain abstain other other