Data sharing restrictions are common in NLP, especially in the clinical domain, but there is limited research on adapting models to new domains without access to the original training data, a setting known as source-free domain adaptation. We take algorithms that traditionally assume access to the source-domain training dataactive learning, self-training, and data augmentationand adapt them for source-free domain adaptation. Then we systematically compare these different strategies across multiple tasks and domains. We find that active learning yields consistent gains across all SemEval 2021 Task 10 tasks and domains, but though the shared task saw successful self-trained and data augmented models, our systematic comparison finds these strategies to be unreliable for source-free domain adaptation. Deep neural networks achieve high performance in many tasks, but typically require annotated training data for each new domain. Domain adaptation algorithms aim to take models trained on one domain (the source domain) and transfer the model's knowledge to another domain (the target domain). They typically try to do this without a huge amount of annotated data in the target domain. Domain adaptation can be easy if the source and target domain have similar distributions, but domains often differ substantially (Wilson and Cook, 2020). While there has been much progress in domain adaptation methods (Kouw, 2018) and even in unsupervised domain adaptation where there are no target-domain labels (Ramponi and Plank, 2020), most methods assume access to the labeled source data. Yet this assumption is often not satisfied, especially in the clinical domain due to privacy concerns (Laparra et al., 2020). SemEval 2021 Task 10 (Laparra et al., 2021), on source-free domain adaptation, called attention to this challenging but more realistic scenario where labeled source data are not accessible, only the model trained on the source domain data can be shared 1 , and little or no labeled target data are available. Participants explored methods including self-training, active learning, and data augmentation (Laparra et al., 2021) but it is hard to make fair comparisons between algorithms since different teams varied in their base implementations. 1. The first systematic comparison of self-training, active learning, and data augmentation for source-free domain adaptation, carried out across multiple tasks and domains. 2. We identify a formulation of source-free active learning that consistently improves performance of the source-domain model, and sometimes even outperforms fine-tuning on a large set of labeled target domain data. 3. We perform an error analysis across tasks and domains and show that the selected formulation of active learning corrects several types of errors that self-training does not. Our code is publicly available. 2 2 Related Work 2.1 Source-free Domain Adaptation Recently, there is rising interest in computer vision to develop methods for unsupervised source-free domain adaptation. Several works utilize a generative framework with a classifier trained on source data to generate labeled training examples (Kurmi et al., 2021; Li et al., 2020) or transfer the target ex-1 In general, it is easier to distribute models than raw data. For example, Lehman et al. (2021) found that none of the algorithms they tried could effectively recover protected health information from a pre-trained language model. 2 github.com/xinsu626/ SourceFreeDomainAdaptation 8352 amples to match the source style (Hou and Zheng, 2020; Sahoo et al., 2020). Other works use self-supervised pseudo-labeling. Liang et al. (2020) proposes source hypothesis transfer that freezes the classifier of the source model domain but fine-tunes the encoding of the source model with a goal to reduce the entropy of individual output prediction while maintaining global diversity. They also augment the strategy by self-supervised pseudo labels via the nearest centroid classifier. Kim et al. (2020) select low self-entropy instances as class prototypes and pseudo-label the remaining target instances based on the distance to the class prototypes and progressively update the models on target data in the manner of self-training. Despite of a growing number of computer vision studies on source-free domain adaptation, there is limited NLP research into this challenging but realistic scenario. Though there is partially related research on continual learning (de Masson d'Autume et al., 2019; Sun et al., 2020) and generalization of pre-trained models (Hendrycks et al., 2020), the only work to explicitly test source-free domain adaptation is SemEval 2021 Task 10 (Laparra et al., 2021), which asked participants to perform source-free domain adaptation on negation detection and time expression recognition. A variety of techniques were applied to this task, including active learning, self-training, and data augmentation. However, different techniques were applied by different participants with different baseline models, so the shared task results do not allow us to make fair comparisons between different techniques. In the current article, we implement and then systematically compare these different techniques. Self-training (Yarowsky, 1995; McClosky et al., 2006) trains a model on a labeled dataset L and then iteratively makes predictions (pseudo-labels) on an unlabeled dataset U and re-trains. On each iteration, the examples in U that the model labels with high confidence (silver labels) are added to L , and the model is retrained on the new, larger L . This process is repeated until no more predictions are highly confident. Self-training has been applied to a variety of domain adaptation scenarios (Ruder and Plank, 2018; Yu et al., 2015; Cui and Bollegala, 2019), but always with the assumption that the original labeled data L is available at each iteration. In source-free domain adaptation, L is not available, so source-free self-training could train on only the pseudo-labels, and it is unclear whether that would yield a superior or inferior model. Active learning selects a small number of examples to be manually annotated, using strategies designed to select the examples that should most benefit the model. Various active learning selection strategies have been developed (see the survey of Settles, 2009), and recent work has shown the benefits of active learning even with pre-trained transformer models (Ein-Dor et al., 2020). Active learning is also frequently used in domain adaptation. For example, Chan and Ng (2007) applied uncertainty sampling for domain adaptation of word sense disambiguation models, and Rai et al. (2010) combined model confidence and a domain discriminator to select target-domain examples for sentiment analysis. As with self-training, active learning algorithms typically assume that the source-domain training data is available and can be combined with target-domain examples. Thus, the efficacy of source-free active learning is currently unclear. Data Augmentation enhances limited data by using existing resources (WordNet, similar datasets, etc.) and/or rule-based transformations of the training data to create new training examples. A variety of data augmentation techniques have been proposed (see the survey of Liu et al., 2020) including back-translation (Sennrich et al., 2016; Wang et al., 2021), lexical-substitution (Zhou et al., 2019; Arefyev et al., 2020; Wei and Zou, 2019; Miao et al., 2020), noise injection (Wei and Zou, 2019), conditional generation (Juuti et al., 2020; Malan-drakis et al., 2019; Kobayashi, 2018), and data transformation with task-specific rules or templates (Sahin and Steedman, 2018; Wang et al., 2021; Xu et al., 2020). Data augmentation assumes access to the source-domain training data, so cannot be used by itself in source-free domain adaptation. It could be coupled with source-free self-training or source-free active learning, but researchers have not yet systematically explored such combinations. We base our experiments off of the data and source-domain models from the tasks of SemEval 2021 Task 10: negation detection and time expression recognition. We select these tasks because: 1. They represent real-world data-sharing problems: the negation source-domain data cannot currently be distributed and the time expression source-domain data is difficult to gain access to due to the complex data use agreements (La-parra et al., 2021). Only the task organizers had access to the data and permission to distribute models trained on the (de-identified) data. 2. The annotation schemes are complex enough that the problem cannot be easily solved by manually annotating the target domain. Su et al. (2021) found that annotations from annotators given only the time annotation guidelines yielded no gains to models, while annotations from heavily trained annotators did yield gains. 3. These two tasks suffer a large performance loss under domain shift: the source-trained model is 15+ points of F1 lower on the target test set than on the source test set (Laparra et al., 2021). The popular Amazon reviews sentiment analysis dataset (Blitzer et al., 2007) violates the points above: labeled source and target data are easily available, the annotation scheme is easy (it is artifi-cially balanced and removes reviews with neutral labels, as others have noted (He et al., 2018; Miller, 2019)), and the source domain model performs well on the target domain (within 0-4 points of F1). We nonetheless include some experiments on this dataset in appendix A.3. We find that with simple data preprocessing and source-domain hyperparameter tuning, the source-domain model alone outperforms all domain adaptation models from Ye et al. (2020) and Ben-David et al. (2020). the goal is to predict that diarrhea is negated by its context. The source-domain negation detection model was trained on Mayo clinic clinical notes. The target domains are Partners HealthCare clinical notes from the i2b2 2010 Challenge and Beth Israel ICU progress notes from the MIMIC III corpus. SemEval 2021 Task 10 time expression recognition is a sequence-tagging task. The goal is to identify the time entities in the document and label them with SCATE types (Bethard and Parker, 2016). For example, given the sentence: the patient underwent appendicitis surgery on August 29, 2018 , the goal is to label August as Month-Of-Year , 29 as Day-Of-Month , and 2018 as Year . The source-domain time expression recognition model was trained on the Mayo Clinic clinical notes of SemEval 2018 Task 6 (Laparra et al., 2018). The target domains are news articles (also from SemEval 2018 Task 6) and reports from food security warning systems including the UN World Food Programme and the Famine Early Warning Systems Network. Each task has a model trained from a source domain and a test set for each of two target domains. For each target domain, we split the data into 20% as a development set and 80% as a test set. Detailed data information is shown in table 1. Source data We do not use source domain data. We use only the English RoBERTa-base models (Liu et al., 2019) (approx. 125M parameters) that the task organizers fine-tuned on the source domain data sets via the Huggingface Transform-8354 ers library v3.5.1 (Wolf et al., 2020). Target development data We use the development data for fine-tuning the model. For active learning, to simulate manual annotation, we fine-tune on a small number of automatically selected labeled examples. For self-training, no labels are used; we fine-tune on predictions (pseudo-labels) generated by the model on the development data. For oracle experiments, we fine-tune the model on all labeled examples in the development set. Target test data We evaluate on the test data. No fine-tuning is performed. Models always treat this data as unlabeled 3 . Its labels are used only during evaluation. We use the same evaluation metrics as in SemEval 2021 Task 10: precision, recall, and F1 score. We aim for a systematic analysis of three strategies with many different implementations in SemEval 2021 Task 10: self-training, active learning, and data augmentation. Our research questions are: 1. How much can we gain from having human intervention (active learning) and not just the model alone (self-training)? 2. For active learning, given a fixed annotation budget, is it better to do several iterations of selecting examples for annotation and retraining the model, or to select and retrain just once? 3. For self training, given a fixed confidence threshold, is it better to do several iterations of generating pseudo-labels and retraining the model, or to generate and train only once? 4. In each iteration of active learning or self-training, should we use the training data from the previous iteration or start anew? 5. In each iteration of active learning or self-training, should we continue training the model from the previous iteration or the model from the source-domain? 6. Do active learning and self-training improve with data augmentation or work better alone? We design source-free variants of self-training, active learning, and data augmentation that incorporate the following parameters, allowing us to investigate the questions above. self-training or active learning SD the data construction strategy: KeepData to keep the training data from the previous iteration, or ResetData to start anew on each iteration. SM the model training strategy: KeepModel to continue training the model from the previous iteration, or ResetModel to continue training from the source-domain model. SA whether or not to use data augmentation. Algorithm 1 presents our self-training algorithm. It follows standard self-training (Yarowsky, 1995) in using the model to add pseudo-labels to the unlabeled data (line 10). However, there is no source-domain labeled data, so the model can fine-tune only on the pseudo-labels. The remainder of the code ensures that models and/or data are kept, reset, or augmented as per the selected strategies. Self-training requires a measure of model confidence on each prediction. In both tasks, we add pseudo-labeled training data a sentence at a time, so we measure confidence at the sentence level. In negation detection, we use the predicted probability 8355 Algorithm 2: Source-Free Active Learning Algorithm Input: M : the source-domain model D : the development set of the target domain T : the maximum number of iterations K : the number of annotations per iteration SD : the data construction strategy SM : the model training strategy SA : the data augmentation strategy 1 M 0  Copy ( M ) 2 D 0  Copy ( D ) 3 L   4 for i  0 to T do 5 if SD = ResetData then 6 L =  7 D = D 0 8 DU  [ d for d  D sorted by uncertainty of M ( d )] 9 LU  { ( d, Annotate ( d )) for d  top K of DU } 10 L  L  LU 11 if SD = KeepData then 12 D  D  { d for ( d, l )  LU } 13 if SA = Augment then 14 L  L  Augment ( LU ) ; 15 if SM = ResetModel then 16 M  M 0 17 Fine-tune M on L ; at RoBERTa's special sentence-initial token <s>. In time expression recognition, we use the average of the predicted probabilities of the most probable class of each token. Algorithm 2 presents our active learning algorithm. It follows an approach similar to Su et al. (2021). Like most active learning algorithms, the core is to select examples the model is uncertain of (line 8) and then manually annotate them (line 9). Since our development sets are already annotated, we simulate annotation by simply revealing the (previously hidden) labels for the selected examples. Active learning requires a measure of model uncertainty on each prediction. In both tasks, we add annotations a sentence at a time, so we measure uncertainty at the sentence level. In negation detection, we use the predicted entropy at RoBERTa's special sentence-initial token, <s>. In time expression recognition, we use the average of the predicted entropies of the tokens in the sentence. Inspired by Miao et al. (2020), we use a pool-based data augmentation method to automatically increase the size of the training set. In negation detection, we construct a pool of all event words in the unlabeled target domain test data. For each development data example to be augmented, we substitute its event with n randomly-sampled words from the pool. For example, if data augmentation is performed on the sentence: Has no <e> diarrhea </e> , we replace the diarrhea with random words from the pool, resulting in sentences like Has no <e> asthma </e> . In time expression recognition, we construct a pool of words for each time entity type using the guidelines of the SCATE annotation schema, excluding words that do not appear in the unlabeled target domain test data. For each entity in a development data example to be augmented, we substitute it with n randomly-sampled words from the pool for its entity type. For example, in the sentence, the patient underwent appendicitis surgery on August 29, 2018 , there are three time entities (Au-gust: Month-Of-Year, 29: Day-Of-Month, 2018: Year). Data augmentation can therefore generate up to n  3 sentences with different years, months, and days, e.g., the patient underwent appendicitis surgery on September 1st, 2017 . The input to the source-domain models for both tasks is a sentence. The output for the negation detection model is a sentence label (negated or not negated). The output for the time expression model is one label per token (its time entity type). For both tasks, we use the conventional RoBERTa input format, surrounding the sentence with the special tokens <s> and </s>. The negation detection data is already split into sentences. For the time recognition data, we split it into sentences using the English sentencizer from Spacy v2.3.2 (Honnibal et al., 2020). When we fine-tune the source-domain model on the target domain, we keep the same training hyperparameters as were used when the shared task organizers trained the models on the source domains. In source-free domain adaptation, there is no (or very little) labeled development data available, so it is not possible to tune hyperparameters. All hyperparameters are given in appendix A.1. All experiments are run on a single Nvidia P100 GPU. The total approximate GPU hours are 70 hours. In self-training, we set the threshold  to 0.95, and experiment with running just a single iteration and with running 30 iterations with the different 8356 SD and SM strategies. The threshold and the number of iterations are adapted from Su et al. (2021). Training may run for fewer iterations when the stopping conditions are met. In active learning, we set our annotation budget to 96 sentences, and experiment with spending these 96 sentences at once and in 8 iterations with the different SD and SM strategies. For all experiments, we run one version with data augmentation (with n = 5 ) and one without. 1. Source-Domain Model : The baseline. It is unadapted, trained only on the source domain. 2. Fine-Tuned Source-Domain Model : The oracle. It is fine-tuned on the target domain using the entire labeled development set. 3. Self-Distilled Model : A RoBERTa-base model fine-tuned on the development set using pseudo labels generated by the source-domain model. 4. Passive Learning Model : The source-domain model fine-tuned on 96 randomly sampled examples from the labeled development set. Tables 2 and 3 show the results of our experiments. We are interested less in the best model for a particular configuration, but rather in which config-urations are successful across multiple tasks and domains. This is because in source-free domain adaptation, there is typically no (or very little) labeled target domain data available for hyperparameter tuning. Therefore, what we need is a universal strategy that does not require careful tuning. For source-free active learning, we find that even small amounts of annotated data are useful, and that smart data selection (e.g., using uncertainty scores) is usually helpful. The active learning KeepData models (rows 6, 8, 11, and 13 in tables 2 and 3) have higher F1s than the baseline source domain models across all tasks and domains (0.054 F1 higher on average). Active learning KeepData models also outperform passive learning models (that randomly select data) in 14 out of 16 cases, and are at least as good as, and typically much better than, the self-training models (rows 15-24 in tables 2 and 3). The ResetModel+ResetData models always have the worst F1s of the active learning models (rows 7 and 12 in tables 2 and 3). Several active learning models achieve higher F1s than the oracle model that fine-tuned on the full labeled development set (row 8, 10, 11, 13, 14 in table 3 Time: News and row 8, 11, 14 in table 3 Time: Food). 1. If there is sufficient expertise to label the data, use active learning and iteratively adapt the model with the KeepModel+KeepData strategy instead of spending the annotation budget all at 8357 Negation: MIMIC-III Negation: i2b2 # Strategy F P R F P R 1 Source-Domain Model (baseline) 0.656 0.921 0.510 0.837 0.855 0.820 2 Fine-Tuned Source-Domain Model (oracle) 0.868 0.875 0.862 0.925 0.928 0.922 3 Self-Distilled Model 0.623 0.825 0.501 0.846 0.849 0.842 4 Passive Learning Model 0.722 0.792 0.663 0.882 0.914 0.853 Active Learning 5 AL ( 96  1 ) 0.759 0.901 0.656 0.886 0.943 0.836 6 AL ( 12  8 ) + ResetModel + KeepData 0.800 0.828 0.774 0.891 0.951 0.838 7 AL ( 12  8 ) + ResetModel + ResetData 0.618 0.842 0.489 0.778 0.972 0.649 8 AL ( 12  8 ) + KeepModel + KeepData 0.817 0.867 0.773 0.859 0.852 0.865 9 AL ( 12  8 ) + KeepModel + ResetData 0.777 0.890 0.689 0.877 0.928 0.831 Active Learning + Data Augmentation 10 AL ( 96  1 ) + DA (5) 0.708 0.652 0.773 0.883 0.937 0.834 11 AL ( 12  8 ) + ResetModel + KeepData + DA (5) 0.805 0.803 0.806 0.891 0.960 0.831 12 AL ( 12  8 ) + ResetModel + ResetData + DA (5) 0.586 0.489 0.730 0.817 0.960 0.710 13 AL ( 12  8 ) + KeepModel + KeepData + DA (5) 0.805 0.878 0.744 0.881 0.925 0.841 14 AL ( 12  8 ) + KeepModel + ResetData + DA (5) 0.745 0.882 0.645 0.889 0.929 0.852 Self-training 15 ST (1) 0.677 0.916 0.537 0.854 0.871 0.838 16 ST (30) + ResetModel + KeepData 0.679 0.937 0.533 0.857 0.876 0.839 17 ST (30) + ResetModel + ResetData 0.695 0.912 0.562 0.861 0.880 0.843 18 ST (30) + KeepModel + KeepData 0.664 0.906 0.525 0.864 0.890 0.840 19 ST (30) + KeepModel + ResetData 0.654 0.879 0.521 0.858 0.883 0.834 Self-training + Data Augmentation 20 ST (1) + DA (5) 0.654 0.943 0.501 0.863 0.894 0.833 21 ST (30) + ResetModel + KeepData + DA (5) 0.000 0.000 0.000 0.861 0.887 0.838 22 ST (30) + ResetModel + ResetData + DA (5) 0.000 0.000 0.000 0.864 0.897 0.834 23 ST (30) + KeepModel + KeepData + DA (5) 0.000 0.000 0.000 0.854 0.869 0.839 24 ST (30) + KeepModel + ResetData + DA (5) 0.000 0.000 0.000 0.855 0.885 0.827 Table 2: Performance of domain adaptation strategies on the negation detection target domains. This emphasizes a challenge of source-free domain adaptation: more data is not always better data. Since we do not have access to the source domain training data, if we fine-tune on too much target domain data the model may start to forget what it learned on the source domain, i.e., catastrophic forgetting (McCloskey and Cohen, 1989). In these cases, the active learning models, by selecting a small set of just the most uncertain examples, reap the benefits of knowing something about the target domain without losing what they learned from the source domain. For source-free self-training, we find that iteratively updating both model and data is slightly above baseline, and that it is better to start from the source-domain model than from RoBERTa without fine-tuning. The KeepModel+KeepData (without data augmentation) is slightly above the source-domain model across all tasks and domains (0.013 F1 higher on average). Every other configuration, even if they outperform KeepModel+KeepData in one task or domain, is below the source-domain baseline in another. All self-trained models without data augmentation (which start from the source-domain model) do at least outperform self-distilled models (which start from the RoBERTa model without fine-tuning; row 3 in tables 2 and 3). The small gains from the only self-training configu-ration that consistently outperformed the source-domain model suggest that self-training may not be worthwhile for source-free domain adaptation. Data augmentation helped in some cases (e.g., self-training time expression recognition on news), and hurt in others (e.g., self-training time expression recognition on food security). Data augmentation sometimes led to ill-behaving models: on the negation MIMIC-III dataset, data augmentation made the self-trained model predict all examples as not negated resulting in 0.000 F1 (rows 21 -24 in table 2: Negation-MIMIC-III). This suggests that data augmentation (or at least the variants of it that we explored) is probably not viable for source-free domain adaptation where no labeled data for tuning strategies is available. once. This is the best model without data augmentation in three of the four domains (Nega-tion: MIMIC III, Time: News, Time: Food). Note that expertise is important: Su et al. (2021) found that active learning with non-experts in the face of a complex annotation scheme did not yield performance improvements. 2. Self-training and data augmentation, at least as implemented here, are not good choices for source free domain adaptation: sometimes they led to gains, and sometimes they led to losses. While a good strategy could be found by labeling some target domain data and performing hyperparameter search, such annotation effort would have a higher payoff if used for active learning instead. 3. Active learning is better than passive learning: smart example selection is better than random example selection. 4. Self-training is better than self-distillation: the models benefit from the task knowledge learned from the source-domain. Our systematic analysis allowed us to make the above more specific suggestions than the shared task's main suggestion that the best performing [systems] incorporated. . . active-learning, handcrafted heuristics or semiautomatically building a training set (Laparra et al., 2021). We performed an error analysis to try to determine if different adaptation strategies resulted in different types of errors being corrected (as compared to the source domain model). For negation detection we sampled and categorized around 200 errors of the source-domain model for each target domain. When the model failed to predict a negation, we manually categorized the error by the negation cue ( no , free , absent , etc.). When the model predicted a negation it should not have, we manually cate-8358 Time: News Time: Food # Strategy F P R F P R 1 Source-Domain Model (baseline) 0.771 0.772 0.770 0.781 0.834 0.734 2 Fine-Tuned Source-Domain Model (oracle) 0.844 0.826 0.864 0.851 0.841 0.861 3 Self-Distilled Model 0.572 0.590 0.555 0.766 0.831 0.711 4 Passive Learning Model 0.796 0.783 0.809 0.770 0.755 0.785 Active Learning 5 AL ( 96  1 ) 0.812 0.800 0.825 0.819 0.821 0.818 6 AL ( 12  8 ) + ResetModel + KeepData 0.812 0.794 0.830 0.842 0.844 0.840 7 AL ( 12  8 ) + ResetModel + ResetData 0.771 0.771 0.770 0.781 0.832 0.737 8 AL ( 12  8 ) + KeepModel + KeepData 0.861 0.844 0.879 0.872 0.866 0.879 9 AL ( 12  8 ) + KeepModel + ResetData 0.772 0.758 0.787 0.781 0.797 0.765 Active Learning + Data Augmentation 10 AL ( 96  1 ) + DA (5) 0.856 0.829 0.884 0.840 0.824 0.855 11 AL ( 12  8 ) + ResetModel + KeepData + DA (5) 0.860 0.830 0.893 0.856 0.840 0.873 12 AL ( 12  8 ) + ResetModel + ResetData + DA (5) 0.790 0.748 0.836 0.793 0.782 0.805 13 AL ( 12  8 ) + KeepModel + KeepData + DA (5) 0.849 0.820 0.881 0.841 0.821 0.863 14 AL ( 12  8 ) + KeepModel + ResetData + DA (5) 0.853 0.828 0.879 0.856 0.831 0.881 Self-training 15 ST (1) 0.753 0.733 0.774 0.777 0.807 0.750 16 ST (30) + ResetModel + KeepData 0.786 0.791 0.782 0.780 0.815 0.747 17 ST (30) + ResetModel + ResetData 0.727 0.688 0.770 0.787 0.815 0.761 18 ST (30) + KeepModel + KeepData 0.784 0.777 0.792 0.786 0.832 0.745 19 ST (30) + KeepModel + ResetData 0.633 0.551 0.743 0.789 0.829 0.752 Self-training + Data Augmentation 20 ST (1) + DA (5) 0.800 0.794 0.805 0.756 0.787 0.726 21 ST (30) + ResetModel + KeepData + DA (5) 0.789 0.790 0.788 0.754 0.780 0.730 22 ST (30) + ResetModel + ResetData + DA (5) 0.795 0.792 0.798 0.765 0.788 0.744 23 ST (30) + KeepModel + KeepData + DA (5) 0.794 0.801 0.788 0.759 0.786 0.734 24 ST (30) + KeepModel + ResetData + DA (5) 0.797 0.791 0.802 0.747 0.771 0.724 Table 3: Performance of domain adaptation strategies on the time expression recognition target domains. gorized the error into wrong cue (there was a negation cue in the sentence but it did not apply to the target event) or short sentence (especially on the i2b2 domain, the model liked to predict all short sentences as negated). For time expression recognition, we categorized all errors of the source-domain model by entity type (insideoutsidebeginning format) for each target domain. For both tasks, we then calculated how many of these source-domain model errors the best adapted models continued to make. Heatmaps of these analyses are plotted in appendix A.2. Across all tasks and domains, we see that the best self-trained models correct errors roughly evenly across source-domain error categories, while the best active learning models correct different errors, more like the oracle (target-fine-tuned) model. For example, the oracle model and active learning adapted models correct many more wrong cue errors in the negation i2b2 domain, more denies and none errors in the negation MIMIC III domain, more B-Period and B-Month-Of-Year entities in the time news domain, and more B-Season-Of-Year, I-Season-Of-Year, and B-This entities in the time food domain. Some error types appear to be only learnable with substantially more data. Only the oracle model is able to correct errors with the non and afebrile negation cues in the i2b2 domain and with the hold negation cue in MIMIC-III domain. This suggests that the source-domain model may be very con-fident in some types of wrong examples causing them not to be selected in active learning and generating poor pseudo-labels in self-training. In this paper, we present a detailed comparison of the use of active learning, self-training and data augmentation to adapt a source-domain model on a target domain when the source-domain training data is unavailable. We identify a specific formula-8359 tion of source-free active learning that consistently improves performance of the source-domain model. We believe our work highlights the interesting challenges of source-free domain adaptation, and its systematic comparison provides a solid base for future research in this area. Research reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Numbers R01LM012918 and R01LM010090. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Our comparison experiments and proposed formulation are intended to encourage model sharing in source-free domain adaptation while avoiding the risk of privacy leakage caused by direct data sharing. The data we use in this experiment are publicly available and from a shared task, however some of that data is from health institutions and requires a data use agreement to work with the data. Though recent research has found it difficult to recover protected information from trained models (Lehman et al., 2021), there is still some small risk that more complex models may be able to do so. However, as our research is a comparative study, we are not directly releasing models, and thus not risking any release of protected health information.
abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain method method method method abstain abstain method abstain method method method method method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain method result result other other abstain method abstain abstain