Abstract Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant literature on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application. We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes  Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applications  1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild  and use it to organise our literature. From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To further help researchers choose appropriate measures to predict drop in performance  an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word representations (CWR) and contrast with the older distributed representations. We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective. Standard machine learning models do not perform well when tested on data from a different target domain. The performance in a target domain largely depends on the domain divergence (Ben-David et al., 2010)  a notion of distance between the two domains. Thus, efficiently measuring and reducing divergence is crucial for adapting models to the new domain  the topic of domain adaptation . Divergence also has practical applications in predicting the performance drop of a model when adapted to new domains (Van Asch and Daelemans, 2010), and in choosing among alternate models (Xia et al., 2020). Given its importance, researchers have invested much effort to define and measure domain divergence. Linguists use register variation to capture varieties in text  the difference between distributions of the prevalent features in two registers (Biber and Conrad, 2009). Other measures include probabilistic measures like H -divergence (Ben-David et al., 2010), information theoretic measures like Jenssen-Shannon and Kullback-Leibler divergence (Plank and van Noord, 2011; Van Asch and Daelemans, 2010) and measures using higher-order moments of random variables like Maximum Mean Discrepancy (MMD) and Central Moment Discrepancy (CMD) (Gretton et al., 2007; Zellinger et al., 2017). The proliferation of divergence measures challenges researchers in choosing an appropriate measure for a given application. To help guide best practices, we first comprehensively review the NLP literature on domain divergences. Unlike previous surveys, which focus on domain adaptation for specific tasks such as machine translation (Chu and Wang, 2018) and statistical (non-neural network) models (Jiang, 2007; Mar-golis, 2011), our work takes a different perspective. We study domain adaptation through the vehicle of domain divergence measures . First, we develop a taxonomy of divergence measures consisting of three groups: Information-Theoretic, Geometric, and Higher-Order measures. Further, to find the most common group used in NLP, we recognise three novel application areas of these divergences  Data Selection, Learning Representations, and Decisions in the Wild and organise the literature under them. We find that Information-Theoretic measures over word distributions are popular for Data Selection and Decisions in the wild, while Higher-order measures over continuous features are frequent for Learning representations. Domain divergence is a major predictor of performance in the target domain. A better domain divergence metric ideally predicts the corresponding performance drop of a model when applied to a target domain  a practical and important component of Decisions in the Wild . We further help researchers identify appropriate measures for predicting performance drops, through a correlation analysis over 130 domain adaptation scenarios and three standard NLP tasks: Part of Speech Tagging (POS), Named Entity Recognition (NER), and Sentiment Analysis and 12 divergence metrics from our literature review. While information-theoretic measures over traditional word distributions are popular in the literature, are higher-order measures calculated over modern contextual word representations better indicators of performance drop? We indeed find that higher-order measures are superior, but traditional measures are still reliable indicators of performance drop. The closest to our work is (Elsahar and Gall, 2019) who perform a correlation analysis. However, they do not compare against different divergence measures from the literature. Comparatively, we consider more tasks and divergence measures. We review the literature from the perspective of domain divergences and their use-cases in NLP. We aid researchers to select appropriate divergence measure that indicate performance-drops, an important application of divergence measures. We devise a taxonomy for domain divergence measures, shown in Figure 1. It contains three main classes. Individual measures belong to a single class, where relationships can exist between measures from different classes. We provide detailed description of individual measures in Appendix A. Geometric measures calculate the distance between two vectors in a metric space. As a divergence measure, they calculate the distance between features ( tf.idf , continuous representations, etc.) extracted from instances of different domains. The P-norm is a generic form of the distance between two vectors, where Manhattan (p=1) and Euclidean distance (p=2) are common. Cosine (Cos) uses the cosine of the angle between two vectors to measure similarity and 1-Cos measures distance. Geometric measures are easy to calculate, but are ineffective in a high dimensional space as all distances appear the same (Aggarwal et al., 2001). Information-theoretic measures captures the distance between probability distributions. For example, cross entropy over n-gram word distributions are extensively used in domain adaptation for machine translation. f -divergence (Csiszr, 1972) is a general family of divergences where f is a convex function. Different formulations of the f function lead to KL and JS divergence. Chen and Cardie (2018) show that reducing f -divergence measure is equivalent to reducing the PAD measures (see next section). Another special case of f -divergence is the family of  divergences, where KL-Div is a special case of  divergence. Renyi Divergence is a member of the  -divergences and tends towards KL-Div as   1 (Edge A (cid:13) ); Often applied to optimal transport problems, Wasserstein distance measures the amount of work needed to convert one probability distribution to the other as distance and is used extensively for domain adaptation. KL-Div is also related to Cross Entropy (CE). In this paper, CE refers to measures based on entropy. Higher-Order measures consider matching higher order moments of random variables or divergence in a projected space. Their properties are amenable to end-to-end learning based domain adaptation and recently have been extensively adopted. Maximum Mean Discrepancy (MMD) is one such measure which considers matching first order moments of variables in a Reproducible Kernel Hilbert Space. On the other hand, CORAL (Sun et al., 2017) considers second order moments and CMD (Zellinger et al., 2017) considers higher order moments. CORAL and CMD are desirable because they avoid computationally expensive kernel matrix computations. KL-Div can also be considered as matching the first-order moment (Zellinger et al., 2017); Edge B (cid:13) . Proxy-A-Distance (PAD) measures the distance between source and target distributions via the error of a classifier in target domain samples as source domain samples (Ben-David et al., 2007). A few other measures do not have ample support in the literature. These include information-theoretic measures such as Bhattacharya coeffi-cient, higher-order measures like PAD* (Elsahar and Gall, 2019), Word Vector Variance (WVV), and Term Vocabulary Overlap (TVO) (Dai et al., 2019). Our taxonomy synthesises the diversity and Figure 1: Taxonomy for divergence measures. the prevalence of the divergence measures in NLP. Our key observation of the literature is that there are three primary families of applications of divergences (cf. Table 1 in the appendix): ( i ) Data Selection : selects a subset of text from a source domain that shares similar characteristics as target domain. The selected subset is then used to learn a target domain model. ( ii ) Learning Representations : aligns source and target domain distributions and learn domain-invariant representations. ( iii ) Decisions in the Wild : helps practitioners predict the performance or drops in performance of a model in a new target domain. We limit the scope our survey to works that focus on divergence measures. We only consider unsupervised domain adaptation (UDA)  where there is no annotated data available in the target domain. It is more practical yet more challenging. For a complete treatment of neural networks and UDA in NLP, refer to (Ramponi and Plank, 2020). Also, we do not treat multilingual work. While cross-lingual transfer can be regarded as an extreme form of domain adaptation, measuring the distance between languages requires different divergence measures, outside our purview. Divergence measures are used to select a subset of text from the source domain that shares similar characteristics to the target domain. Since the source domain has labelled data, the selected data serves as supervised data to train models in the target domain. We note that the literature pays closer attention to data selection for machine translation compared to other tasks. This can be attributed to its popularity in real-world applications and the difficulty of obtaining parallel sentences for every pair of language. Simple word-level and surface-level text features like word and n-gram frequency distributions and tf.idf weighted distributions have sufficient power to distinguish between text varieties and help in data selection. Geometric measures like cosine, used with word frequency distributions, are effective for selecting data in parsing and POS tagging (Plank and van Noord, 2011). Instead of considering distributions as (sparse) vectors, one can get a better sense of the distance between distributions using information-theoretic measures. Remus (2012) find JS-Div effective for sentiment analysis. While word-level features are useful to select supervised data for an end-task, they also can be used to select data to pre-train language-models subsequently used for NER. Dai et al. (2019) use Term Vocabulary Overlap for selecting data for pretraining language models. Geometric and Information-theoretic measures with word level distributions are inexpensive to calculate. However, the distributions are sparse and continuous word distributions help in learning denser representations. Continuous or distributed representations of words, such as CBOW, Skip-gram (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), address shortcomings of representing text as sparse, frequency-based probability distributions by transforming them into dense vectors learned from freeform text. A geometric measure (e.g., Word Vector Variance used with static word embeddings) is useful to select pre-training data for NER (Dai et al., 2019). Such selected data is found to be similar in tenor (the participants in a discourse, the relationships between them, etc.) to the source data. But static embeddings do not change according to the context of use. In contrast, contextual word representations (CWR)  mostly derived from neural networks (Devlin et al., 2019; Peters et al., 2018) Paper Task(s) Information-Theoretic Geometric Higher-Order Others KL JS Renyi CE Wass. capture contextual similarities between words in two domains. That is, the same word used in two domains in different contexts will have different embeddings. CWRs can be obtained from hidden representations of pretrained neural machine translation (NMT) models. (McCann et al., 2017) have found such representations along with P-norm effective for data selection in MT (Wang et al., 2017). Compared to representations from shallow NMT models, hidden representations of deep neural network language models (LM) like BERT have further improved data selection for NMT (Aharoni and Goldberg, 2020). Divergences can be measured by comparing the probabilities of a language model, in contrast to directly using its hidden representations. If a LM trained on the target domain assigns high probability to a sentence from the source domain, then the sentence should have similar characteristics to the target domain. Cross Entropy (CE) between probability distributions from LMs capture this notion of similarity between two domains. They have been extensively used for data selection in statistical machine translation (SMT) (Yasuda et al., 2008; Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Liu et al., 2014). However, CE based methods for data selection are less effective for neural machine translation (van der Wees et al., 2017; Silva et al., 2018). Instead, van der Wees et al. (2017) come up with a dynamic subset selection where new subset is chosen every epoch during training. We note again the common refrain that sufficient amount of data should be available; here, to train good language models in the target domain. Similar to language models, probabilistic scores from classifiers  which distinguish between samples from two domains  can aid data selection. The probabilities assigned by such classifiers in construing source domain text as target domain has been used as a divergence measures in machine translation (Chen and Huang, 2016). However, the classifiers require supervised target domain data which is not always available. As an alternative, Chen et al. (2017) train a classifier and selector in an alternating optimisation manner. From this literature review, we find that distinct measures are effective for different NLP tasks. Ruder and Plank (2017) argue that owing to their varying task characteristics, different measures should apply. They show that learning a linear combination of measures is useful for NER, parsing and sentiment analysis. However, this is not always possible, especially in unsupervised domain adaptation where there is no supervised data in target domain. We observe that information theoretic measures and geometric measures based on frequency distributions and continuous representations are common for text and structured prediction tasks (cf. Table 1 in the appendix). The effectiveness of higher order measures for these tasks are yet to be ascertained. Further, we find that for SMT data selection, variants of Cross Entropy (CE) measures are used extensively. However, the conclusions of van der Wees et al. (2017) are more measured regarding the benefits of CE and related measures for NMT. Contextual word representations with cosine similarity has found some initial exploration for neural machine translation (NMT), with higher order measures yet to be explored for data selection in NMT. One way to achieve domain adaptation is to learn representations that are domain-invariant which are sufficiently powerful to perform well on an end task (Ganin et al., 2015; Ganin and Lempitsky, 2015). The theory of domain divergence (Ben-David et al., 2010) shows that the target domain error is bounded by the source domain error and domain divergence ( H -divergence) and reducing the domain divergence results in domain-invariant representation. The theory also proposes a practical alternative to measure H -divergence called PAD. The idea is to learn a representations that confuses a domain discriminator sufficiently to make samples from two domains indistinguishable. Ganin et al. (2015) operationalise PAD in a neural network named Domain Adversarial Neural Networks (DANN). The network employs a minmax game  between the representation learner and the domain discriminator  inspired by Generative Adversarial Networks (Goodfellow et al., 2014). The representation learner is not only trained to minimise a task loss on source domain, but also maximise a discriminator's loss, by reversing the gradients calculated for the discriminator. Note that this does not require any supervised data for target domain. In later work, Bousmalis et al. (2016) argue that domain-specific peculiarities are lost in a DANN, and propose Domain Separation Networks (DSN) to address this shortcoming. In DSN, both domain-specific and -invariant representations are captured in a sharedprivate network. DSN is flex-ible in its choice of divergence measures and they find PAD performs better than MMD. Here, we limit our review to works utilising divergence measures. We exclude feature-based UDA methods such as Structural Corresponding Learning (SCL) (Blitzer et al., 2006), Autoencoder-SCL and pivot based language models (Ziser and Reichart, 2017, 2018, 2019; Ben-David et al., 2020). Obtaining domain invariant representations is desirable for many different NLP tasks, especially for sequence labelling where annotating large amounts of data is hard. They are typically used when there is a single source domain and a single target domain  for sentiment analysis (Ganin et al., 2016), NER (Zhou et al., 2019), stance detection (Xu et al., 2019), machine translation (Britz et al., 2017; Zeng et al., 2018). The application of DANN and DSN to a variety of tasks are testament of their generality. DANN and DSN are applied in other innovative situations. Text from two different periods of time can be considered as two different domains for intent classification (Kim et al., 2017). Gui et al. (2017) consider clean formal newswire data as source domain and noisy, colloquial, unlabeled Twitter data as the target domain and use adversarial learning to learn robust representations for POS. Commonsense knowledge graphs can help in learning domain-invariant representations as well. Ghosal et al. (2020) condition DANN with an external commonsense knowledge graph using graph convolutional neural networks for sentiment analysis. In contrast, Wang et al. (2018) use MMD outside the adversarial learning framework. They use MMD to learn to reduce the discrepancy between neural network representations belonging to two domains. Such concepts have been explored in computer vision (Tzeng et al., 2014). While single source and target domains are common, complementary information available in multiple domains can help to improve performance in a target domain. This is especially helpful when there is no large-scale labelled data in any one domain, but where smaller amounts are available in several domains. DANN and DSN have been extended to such multi-source domain adaptation: for intent classification (Ding et al., 2019), sentiment analysis (Chen and Cardie, 2018; Li et al., 2018; Guo et al., 2018; Wright and Augenstein, 2020) and machine translation (Gu et al., 2019; Wang et al., 2019). DANN and DSN can also help in multitask learning which considers two complementary tasks (Caruana, 1997). A key to multitask learning is to learn a shared representation that captures the common features of two tasks. However, such representations might still contain task-specific information. The shared-private model of DSN helps in disentangling such representations and has been used for sentiment analysis (Liu et al., 2017), Chinese NER and word segmentation (Cao et al., 2018). Also, although beyond the scope of our discussion here, DANN and DSN have been used to learn language-agnostic representations for text classification and structured prediction in multilingual learning (Chen et al., 2018; Zou et al., 2018; Ya-sunaga et al., 2018). Most works that adopt DANN and DSN framework reduce either the PAD or MMD divergence. However, reducing the divergences, combined with other auxiliary task specific loss functions, can result in training instabilities and vanishing gradients when the domain discriminator becomes increasingly accurate (Shen et al., 2018). Using other higher order measures can result in more stable learning. In this vein, CMD has been used for sentiment analysis (Zellinger et al., 2017; Peng et al., 2018), and Wasserstein distance has been used for duplicate question detection (Shah et al., 2018) and to learn domain-invariant attention distributions for emotional regression (Zhu et al., 2019). The review shows that most works extend the DSN framework to learn domain invariant representations in different scenarios (cf. Table 1, in the ap-pendix). The original work from (Bousmalis et al., 2016) includes MMD divergence besides PAD, which is not adopted in subsequent works, possibly due to the reported poor performance. Most works require careful balancing between multiple objective functions (Han and Eisenstein, 2019), which can affect the stability of training. The stability of training can be improved by selecting appropriate divergence measures like CMD (Zellinger et al., 2017) and Wasserstein Distance (Arjovsky et al., 2017). We believe additional future works will adopt such measures. Models can perform poorly when they are deployed in the real world. The performance degrades due to the difference in distribution between training and test data. Such performance degradation can be alleviated by large-scale annotation in the new domain. However, annotation is expensive, and  given thousands of domains  quickly becomes infeasible. Predicting the performance in a new domain, where there is no labelled data, is thus important. Much recent work provides theory (Rosenfeld et al., 2020; Chuang et al., 2020; Steinhardt and Liang, 2016). As models are put into production in the real world, this application becomes practically important as well. Empirically, NLP considers the divergence between the source and the target domain to predict performance drops. machine learning model in new domains. Information theoretic measures like Renyi-Div and KL-Div has been used for predicting performance drops in POS (Van Asch and Daelemans, 2010) and Cross-Entropy based measure has been used for dependency parsing (Ravi et al., 2008). Prediction of performance can also be useful for machine translation where obtaining parallel data is hard. Based on distance between languages, (Xia et al., 2020) predict performance of the model on new languages for MT, among other tasks. Such performance prediction models have also been done in the past for SMT (Birch et al., 2008; Specia et al., 2013). However, Ponomareva and Thelwall (2012) argue that predicting drops in performance is more appropriate compared to raw performance. They find that JS-Div effective for predicting performance drop of Sentiment Analysis systems. Only recently, predicting model failures in practical deployments from an empirical viewpoint has regained attention. Elsahar and Gall (2019) find the efficacy of higher-order measures to predict the drop in performance for POS and SA and do not rely on hand crafted measures as in previous works. However, analysing performance drops using CWR is still lacking. We tackle this in the next section. A practical use case of domain divergences is to predict the performance drop of a model applied to a new domain. We ask how relevant are traditional measures over word distributions compared to higher-order measures like CMD and MMD over contextual word representations like BERT, Elmo, DistilBERT (Devlin et al., 2019; Peters et al., 2018; Sanh et al., 2019)? We perform an empirical study to assess their suitability to predict performance drops for three important NLP tasks: POS, NER, and SA leaving machine translation to future work. Performance difference between the source and the target domain depends on the divergence between their feature distributions (Ben-David et al., 2010). We assume a co-variate shift, as in (Ganin et al., 2016), where the marginal distribution over features change, but the conditional label distributions does not  i.e., PD s ( y | x ) = PDT ( y | x ) PD s ( x ) (cid:54) = PDT ( x ) . Although difference in conditional label distribution can increase the H Divergence measure (Wisniewski and Yvon, 2019), it requires labels in the target domain for assessment. In this work, we assume no labelled data in the target domain, to best mimic realistic settings. Datasets: For POS, we select 5 different corpora from the English Word Tree Bank of Universal Dependency corpus (Nivre et al., 2016) 1 and also include the GUM, Lines, and ParTUT datasets. We follow Elsahar and Gall (2019) and consider these as 8 domains. For NER, we consider CONLL 2003 (Tjong Kim Sang and De Meulder, 2003), Emerging and Rare Entity Recognition Twitter (Derczynski et al., 2017) and all 6 categories in OntoNotes v5 (Hovy et al., 2006) 2 , resulting in 8 domains. For SA, we follow Guo et al. (2020), selecting the same 5 categories 3 for experiments (Liu et al., 2017). Divergence Measures: We consider 12 divergences. For Cos, we follow the instance based calculation (Ruder et al., 2017). For MMD, Wasserstein and CORAL, we randomly sample 1000 sentences and average the results over 3 runs. For MMD, we experiment with different kernels ( cf. Appendix A) and use default values of  from the GeomLoss package (Feydy et al., 2019). For TVO, KL-div, JS-div, Renyi-div, based on word frequency distribution we remove stop-words and consider the top 10k frequent words across domains to build our vocabulary (Ruder et al., 2017; Guru-rangan et al., 2020). We use  =0.99 for Renyi as found effective by Plank and van Noord (2011). We do not choose CE as it is mainly used in MT and ineffective for classification and structured prediction (Ruder et al., 2017). Model Architecture: For all our experiments, unless otherwise mentioned, we use the pre-trained DistilBERT (Sanh et al., 2019) model. It has competitive performance to BERT, but has faster inference times and lower resource requirements. For every text segment, we obtain the activations from the final layer and average-pool the representations. We train the models on the source domain training split and test the best model  picked from validation set grid search  on the test dataset of the same and other domains (cf. Appendix C). For POS and NER, we follow the original BERT model where a linear layer is added and a prediction is made for every token. If the token is split into 1 Yahoo! Answers, Email, NewsGroups, Reviews and We-blogs. 2 Broadcast News (BN), Broadcast Conversation (BC), Magazine (MZ), Telephone Conversation (TC) and Web (WB). 3 Apparel, Baby, Books, Camera and MR. multiple tokens due to Byte Pair Encoding, the label for the first token is predicted. For SA and domain discriminators, we pool the representation from the last layer of DistilBERT and add a linear layer for prediction (Appendix B). For POS, the PAD measure has the best correlation with performance drop (cf. Table 2). Information-theoretic measures over word frequency distributions, such as JS-div, KL-div, and TVO, which have been prevalent for data selection and performance drop use cases (cf. Table 1) are comparable to PAD. Plank et al. (2014) claim that the errors in POS are dictated by out of vocabulary words. Our findings validate their claim, as we find strong correlation between POS performance drop and word probability distribution measures For NER, MMD-RQ provides the best correlation of 0.495. CORAL  a higher-order measure  and JS-div are comparable. For SA, Renyi-div and other information-theoretic measures provide considerably better correlation compared to higher-order measures. Cos is a widely-used measure across applications, however it did not provide significant correlation for either task. TVO is used for selecting pretraining data for NER (Dai et al., 2019) and as a measure to gauge the benefits of fine-tuning pre-trained LMs on domain-specific data (Gururangan et al., 2020). Although TVO does not capture the nuances of domain divergences, it has strong, reliable correlations for performance drops. PAD has been suggested for data selection in SA by Ruder and Plank (2017) and for predicting drop in performance by Elsahar and Gall (2019). Our analysis confirms that PAD provides good correlations across POS, NER, and SA. We find no single measure to be superior across all tasks. However, information theoretic measures consistently provide good correlations. Currently, when contextual word representations dictate results in NLP, simple measures based on frequency distributions are strong baselines for predicting performance drop. Although higher-order measures do not always provide the best correlation, they are differentiable, thus suited for end-to-end training of domain-invariant representations. Why are some divergence measures better at predicting drops in performance? The one-dataset-one-domain is a key assumption in such works. However, many works have questioned this assumption (Plank and van Noord, 2011). Multiple domains may exist within the same domain (Webber, 2009) and two different datasets may not necessarily be considered different domains (Irvine et al., 2013). Recently Aharoni and Goldberg (2020) show that BERT representations reveal their underlying domains. They qualitatively show that a few text segments from a dataset actually belong to another domain. However the degree to which the samples belong to different domains is unclear. We first test the assumption that different datasets are different domains using Silhouette scores (Rousseeuw, 1987) which quantify the separability of clusters. We initially assume that a dataset is in its own domain. A positive score shows that datasets can be considered as well-separated domains; a negative score shows that most of the points within a dataset can be assigned to a nearby domain; and 0 signifies overlapping domains. We calculate Silhouette scores and t-SNE plots (Maaten and Hinton, 2008) for different divergence measures. Refer to the plots (Figures 3a to 3c) and calculation details in Appendix D. Almost all the measures across different tasks have negative values close to 0 (Table 2, (r)). For POS, CORAL, Wasserstein and Cos strongly indicate that text within a dataset belongs to other Measure Correlations Silhouette Coefficients POS NER SA POS NER SA Cos 0.018 0.223 -0.012  1 . domains. However, for MMD-Gaussian the domains overlap (Figure 2a). For NER, MMD-Gaussian and MMD-Laplacian indicate that the clusters overlap while all other metrics have negative values. For SA, JS-Div has positive values compared to other measures, and as seen in Figure 2c, we can see a better notion of distinct clusters. The Silhouette scores along with the t-SNE plots show that datasets are, in fact, not distinct domains. Considering data-driven methods for defining domains is needed (Aharoni and Goldberg, 2020). If there are indeed separate domains, does it explain why some measures are better than the others? We see better notions of clusters for NER and sentiment analysis ( cf. Figures 2b and 2c). We can expect the drop in performance to be indicative of these domain separations. Comparing the best correlations from Table 2, correlations for NER and sentiment analysis are higher compared with POS. For POS, there are no indicative domain clusters and the correlation between domain divergence and performance may be less; whereas for SA, both the t-SNE plot and the Silhouette scores for JS-Div ( cf. Figure 2c) corroborate comparatively better separation. If datasets are indeed different domains, these divergence measures are reliable indicators of performance drops. If they are not, there might be other confounding factors (such as differences in label distribution) and one has to be cautious in using them. Domain overlap also has consequences for data selection strategies. For example, Moore and Lewis (2010) select pseudo in-domain data from source corpora (cf Section 3.1). As the Silhouette coefficients are negative and close to 0, many data points in a dataset belong to nearby domains. Data selection strategies thus may be effective. If the Silhouette coefficients are more negative and if more points in the source aptly belong to the target domain, we should expect increased sampling from such source domains to yield additional performance benefits in the target domain. We survey domain adaptation works, focusing on divergence measures and their usage for data selection , learning domain-invariant representations , and making decisions in the wild . We synthesised the divergence measures into a taxonomy of information theoretic , geometric and higher-order measures. While traditional measures are common for data selection and making decisions in the wild, higher-order measures are prevalent in learning representations. Based on our correlation experiments, silhouette scores, and t-SNE plots, we make the following recommendations:  PAD is a reliable indicator of performance drop. It is best used when there are sufficient examples to train a domain discriminator. JS-Div is symmetric and a formal metric. It is related to PAD, easy to compute, and serves as a strong baseline. While Cosine is popular, it is an unreliable indicator of performance drop. One-dataset-is-not-one-domain. Instead, cluster representations and define appropriate domains. We would also like to acknowledge the support of the NExT research grant funds,supported by the National Research Foundation,Prime Ministers Of-fice, Singapore under its IRC@ SG Funding Initiative, and to gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX Titan XGPU used in this research.
abstain abstain abstain objective objective method method objective result abstain abstain abstain abstain abstain abstain other abstain objective method method objective objective result abstain abstain method abstain result method abstain method abstain abstain result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method method method method method method method method method method method abstain method method method method abstain abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain other