summaries: This paper conducts a large-scale empirical study of ERM models tested in the domainBed pipeline, together with the empirical results tested, the authors also investigated the numerical values of several different measures, which allows the authors to further study the relationship between multiple different measures and the empirical performances. The empirical study suggests that there may be interesting connections between the target-entropy and the empirical performances. The paper candidly acknowledges its limitations of being purely empirical.  The paper provides a large scale study of out-of-distribution generalization measures. It first provides a brief introduction of the core theoretical foundations of the field. Then the paper trains 12000 models on DomainBed Benchmarks to examine the predictive power of theory-based measures. The paper argues that theory based measure fail to accurately capture the OOD generalization behavior. Following the methodology of Jiang 2019 et al, the paper examine how various empirical measures (sharpness, entropy, Fisher, ...) predict OOD generalization behavior. In the end, the paper lists the predictive measures and examines their performance details. This work tried to find that which measures can be used for predicting the the out-of-domain generalization for the deep neural networks trained using Empirical Risk Minimization (ERM). The authors started from the domain adaptation theory of Ben-David, 2007 but found that it had limited ability. Then they explored many other measures, including Fisher based measures, Jacobian Norm based measures, Mixup based Measures, and entropy on target data. They found the best single factor led to around 0.70 Spearman's ρ and the joint use can boost the value to 0.80. Prior work has found that empirical risk minimization performs unexpectedly well on domain generalization tasks. In this work, the authors try to explain this phenomenon. First, they use theory developed for domain adaptation to bound the target domain test error, finding that theory-based measures do not adequately predict the target domain test error, especially when target domain labels are not available. Next, the authors try a variety of empirical measures. They find that some measures outperform the theory, especially when combined with the in-domain test error.
abstractText: Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goal of this paper is to further understand this phenomenon. In particular, we study the extent to which the seminal domain adaptation theory of Ben-David et al. (2007) explains the performance of ERMs. Perhaps surprisingly, we find that this theory does not provide a tight explanation of the out-of-domain generalization observed across a large number of ERM models trained on three popular domain generalization datasets. This motivates us to investigate other possible measures—that, however, lack theory—which could explain generalization in this setting. Our investigation reveals that measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy are good predictors of the out-of-distribution generalization of ERM models. We hope that our work helps galvanize the community towards building a better understanding of when deep networks trained with ERM generalize out-of-distribution.
authors.name: Ramakrishna Vedantam
authors.name: David Lopez-Paz
authors.name: David J. Schwab
references.authors: S. Amari
references.title: Natural gradient works efficiently in learning
references.venue: Neural Comput.,
references.authors: M. Arjovsky L. Bottou I. Gulrajani D. Lopez-Paz
references.title: Invariant risk minimization
references.authors: S. Arora R. Ge B. Neyshabur Y. Zhang
references.title: Stronger generalization bounds for deep nets via a compression approach
references.authors: P. Bartlett D.J. Foster M. Telgarsky
references.title: Spectrally-normalized margin bounds for neural networks
references.authors: S. Ben-David J. Blitzer K. Crammer F. Pereira
references.title: Analysis of representations for domain adaptation
references.venue: Advances in Neural Information Processing Systems,
references.authors: S. Ben-David J. Blitzer K. Crammer A. Kulesza F. Pereira J. Vaughan
references.title: A theory of learning from different domains
references.venue: Machine Learning,
references.authors: L. Carratino M. Cissé R. Jenatton Vert J.-P
references.title: On mixup regularization
references.authors: C. Cortes M. Mohri A.M. Medina
references.title: Adaptation based on generalized discrepancy
references.venue: J. Mach. Learn. Res.,
references.authors: A. Daniely E. Granot
references.title: Generalization bounds for neural networks via approximate description length
references.authors: H. Drucker Y. Le Cun
references.title: Improving generalization performance using double backpropagation
references.venue: IEEE Trans. Neural Netw.,
references.authors: G.K. Dziugaite D.M. Roy
references.title: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data
references.authors: G.K. Dziugaite A. Drouin B. Neal N. Rajkumar E. Caballero L. Wang I. Mitliagkas D.M. Roy
references.title: In search of robust measures of generalization
references.authors: C. Fang Y. Xu D.N. Rockmore
references.title: Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias
references.venue: IEEE International Conference on Computer Vision,
references.authors: Y. Ganin E. Ustinova H. Ajakan P. Germain H. Larochelle F. Laviolette M. Marchand V. Lempitsky
references.title: Domain-Adversarial training of neural networks
references.authors: M. Ghifary W. Bastiaan Kleijn M. Zhang D. Balduzzi
references.title: Domain generalization for object recognition with multi-task autoencoders
references.authors: A. Gretton A. Smola J. Huang M. Schmittfull K. Borgwardt B. Schölkopf
references.title: Covariate shift by kernel mean matching
references.venue: Dataset shift in machine learning,
references.authors: I. Gulrajani D. Lopez-Paz
references.title: In search of lost domain generalization
references.authors: D. Hendrycks M. Mazeika T. Dietterich
references.title: Deep anomaly detection with outlier exposure
references.authors: D. Hendrycks S. Basart N. Mu S. Kadavath F. Wang E. Dorundo R. Desai T. Zhu S. Parajuli M. Guo D. Song J. Steinhardt J. Gilmer
references.title: The many faces of robustness: A critical analysis of Out-of-Distribution generalization
references.authors: J. Hoffman D.A. Roberts S. Yaida
references.title: Robust learning with jacobian regularization
references.authors: S. Jastrzebski D. Arpit O. Astrand G. Kerg H. Wang C. Xiong R. Socher K. Cho K. Geras
references.title: Catastrophic fisher explosion: Early phase fisher matrix impacts generalization
references.authors: Y. Jiang B. Neyshabur H. Mobahi D. Krishnan S. Bengio
references.title: Fantastic generalization measures and where to find them
references.authors: D. Krueger E. Caballero Jacobsen J.-H A. Zhang J. Binas R. Le Priol A. Courville
references.title: Out-of-Distribution generalization via risk extrapolation (REx)
references.authors: D. Li Y. Yang Song Y.-Z T.M. Hospedales
references.title: Learning to generalize: Meta-Learning for domain generalization
references.authors: D. Li Y. Yang Song Y.-Z T.M. Hospedales
references.title: Deeper, broader and artier domain generalization
references.authors: M. Mohri A.M. Medina
references.title: New analysis and algorithm for learning with drifting distributions
references.authors: B. Neyshabur R. Tomioka N. Srebro
references.title: Norm-Based capacity control in neural networks
references.authors: B. Neyshabur S. Bhojanapalli D. McAllester N. Srebro
references.title: Exploring generalization in deep learning
references.authors: R. Novak Y. Bahri D.A. Abolafia J. Pennington J. Sohl-Dickstein
references.title: Sensitivity and generalization in neural networks: an empirical study
references.authors: S.J. Pan I.W. Tsang J.T. Kwok Q. Yang
references.title: Domain adaptation via transfer component analysis
references.venue: IEEE Trans. Neural Netw.,
references.authors: A. Paszke S. Gross F. Massa A. Lerer J. Bradbury G. Chanan T. Killeen Z. Lin N. Gimelshein L. Antiga A. Desmaison A. Köpf E. Yang Z. DeVito M. Raison A. Tejani S. Chilamkurthy B. Steiner L. Fang J. Bai S. Chintala
references.title: PyTorch: An imperative style, High-Performance deep learning
references.authors: B. Sun K. Saenko
references.title: Deep CORAL: Correlation alignment for deep domain adaptation
references.authors: V.N. Vapnik
references.title: An overview of statistical learning theory
references.venue: IEEE Trans. Neural Netw.,
references.authors: D. Varga A. Csiszárik Z. Zombori
references.title: Gradient regularization improves accuracy of discriminative models
references.authors: Y. Wu E. Winston D. Kaushik Z. Lipton
references.title: Domain adaptation with Asymmetrically-Relaxed distribution alignment
references.venue: Proceedings of the 36th International Conference on Machine Learning,
references.authors: S. Yan H. Song N. Li L. Zou L. Ren
references.title: Improve unsupervised domain adaptation with mixup training
references.authors: C. Zhang S. Bengio M. Hardt B. Recht O. Vinyals
references.title: Understanding deep learning requires rethinking generalization
references.authors: H. Zhang M. Cisse Y.N. Dauphin D. Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.venue: arXiv preprint arXiv:1710.09412,
references.authors: H. Zhang M. Cisse Y.N. Dauphin D. Lopez-Paz
references.title: mixup: Beyond empirical risk minimization
references.authors: H. Zhao R.T. des Combes K. Zhang G.J. Gordon
references.title: On learning invariant representation for domain adaptation
sections.heading: 1 Introduction
sections.text: Conventional wisdom in domain generalization was recently upturned by the work of Gulrajani & Lopez-Paz (2020) on the DomainBed benchmark. In their work, the authors use careful model selection and evaluation to show that models trained using empirical risk minimization (Vapnik, 1999) are able to achieve near state-of-the-art performance on a variety of popular benchmarks, surpassing the performance of most algorithms specialized for domain generalization (Arjovsky et al., 2019; Yan et al., 2020; Li et al., 2017a).
  In a parallel research effort, there has been an increased interest in understanding when deep neural networks trained using empirical risk minimization generalize well in-domain (Jiang et al., 2019; Neyshabur et al., 2017; Dziugaite & Roy, 2017; Bartlett et al., 2017; Arora et al., 2018; Daniely & Granot, 2019). However, understanding when empirical risk minimizers are able to generalize out-of-distribution remains poorly understood. Answering this question would help us characterize the failure modes of domain generalization algorithms, and to develop specialized baselines that would allow us to break through the dominance of empirical risk minimization.
  We argue that the out-of-distribution generalization performance of a classifier depends mainly on two factors. On the one hand, we expect good out-of-distribution generalization only for classifiers able to achieve good in-distribution generalization (also called source-domain validation accuracy). On the other hand, we expect out-of-distribution generalization to degrade as the discrepancy between the distributions of training and testing data increases, relative to the function family of interest. This reveals an interplay between the properties of the neural network, the used training data, and the
  ∗ Both D.L.P and D.J.S. contributed to the work equally 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  differences between training and testing regimes (Figure 1). Given all of these distinct ingredients, how does one build a coherent theory of domain generalization with strong predictive power for modern, over-parametrized neural networks?
  The seminal work of Ben-David et al. (2007, 2010) lays down a theoretical foundation for the related setup of unsupervised domain adaptation. In particular, the theory in Ben-David et al. (2007, 2010) upper-bounds the out-of-distribution performance of an ERM based on a fixed feature space as the sum of two terms: (i) the source-domain validation accuracy, and (ii) the distance between the input distribution of source-domain and target-domain examples under the fixed feature space. This theory has found success in practice, improving both linear models (Ben-David et al., 2007) and deep neural networks (Ganin et al., 2015) in the setup of unsupervised domain adaptation. Algorithmically speaking, one can implement this theory by finding a feature representation under which (i) one can perform well on source-domain inputs, and (ii) it is not possible to distinguish source-domain inputs from target-domain inputs.
  While the theory above deals with the setup of unsupervised domain adaptation (where algorithms have access to unlabeled target-domain data during training), here we are interested in the related but distinct domain generalization scenario where algorithms only have access to source-domain data during training. Therefore, to remain in the domain generalization setup, we will put the theory of Ben-David et al. (2007) to use from a different angle. Instead of training representations in such a way that encourages source-domain and target-domain inputs to be indistinguishable, our methodology consists in training a very large number of ERM models only using source-domain data, to later investigate if the models exhibiting good out-of-distribution generalization indeed learned a feature space under which it is difficult to tell apart source-domain and target-domain inputs.
  Training over 12,000 deep neural network image classifiers using empirical risk minimization, using the standard DomainBed pipeline, we reveal that the theory of Ben-David et al. (2007) has limited power to predict when a classifier will generalize out-of-domain. We investigate this further by unpacking whether assumptions in the theory are violated or if the bounds are slack. Our findings suggest that in practice, while the assumptions from the theory are largely not violated in the learned feature spaces of ERMs, one can obtain good generalization despite being able to tell apart the source and target data, limiting the practical applicability of the theory to explain domain generalization. We investigate and provide more intuitions for this phenomenon in section 4.
  In search of other correlates of domain generalization that might point the way to new theoretical approaches, we set out to build a catalog of measures that could predict the out-of-distribution generalization performance of our suite of ERM trained neural networks. In building such a catalog of generalization measures, we take inspiration from recent literature undertaking a similar effort to explain variance of in-domain generalization of deep neural networks (Jiang et al., 2019; Neyshabur et al., 2017; Dziugaite et al., 2020). After an exhaustive empirical evaluation, we find that several of the proposed measures are relatively strong predictors of out-of-distribution performance. These include measures related to predictive entropy, the Mixup criterion (Zhang et al., 2017), the norm of the Jacobian (Novak et al., 2018), and the Fisher information matrix (Amari, 1998).
  To summarize, our work makes the following contributions:
  • We perform a large scale empirical study testing the theory from Ben-David et al. (2007, 2010) on deep neural networks trained on the DomainBed (Gulrajani & Lopez-Paz, 2020) domain generalization benchmark.
  • We find that the theory has limited ability to predict which ERMs will generalize out-ofdistribution.
  • We perform an empirical evaluation of several other candidate measures for explaining domain generalization of neural networks, finding measures that outperform theoretically motivated quantities.
sections.heading: 2 Background on Relevant Theory
sections.text: There is a rich line of work on the theory of domain adaptation, which focuses on learning and adapting when presented with test data drawn from a different input distribution from that seen during training (Ben-David et al., 2007; Mohri & Medina, 2012; Cortes et al., 2019). There are also a number of algorithms for domain adaptation which make use of unlabeled data to learn more invariant predictors, both for classical kernel machines (Pan et al., 2011; Gretton et al., 2009) as well as deep neural networks (Ganin et al., 2015). The results typically provide bounds on the test error in a target domain in terms of the test error of the model on a training domain, and a measure of the difference between the train and target input distributions in a feature space. Here we are interested in whether the theory can accurately predict out-of-distribution performance of an ERM.
  In particular, we first focus on a classic result from Ben-David et al. (2007), which, given a fixed feature representation (R), derives generalization bounds for the test performance in terms of how well the model can distinguish source from target samples. This work focuses on the single-source (SS) setting, where a single source-domain is used for training. A central result from this work is a bound on the test error on the target domain ( T (h)), consisting of three distinct terms: the sourcedomain validation error ( S(h)), the so-called H-divergence between the source and target domains, and the joint source-target validation error when optimizing a single model for both environments, i.e. λ = minh∈H{ T (h) + S(h)}. In fact, in the theory, λ-closeness is assumed, and we will test this assumption later. We consider the predictive value of each of these three terms separately as well as their sum in our analysis. The H-divergence can be understood as related to and approximated by the best performance of a classifier in the function family H trained to distinguish samples from the source and target domains.
  Subsequent work generalized this theory to multi-source (MS) training in Ben-David et al. (2010). Assuming an equal weighting of all training domains, the gap between the test errors in the target and source domains can be bounded above by the sum of two terms: the average λj , where λj = minh∈H{ T (h) + j(h)}, and the H∆H-divergence, similar to the H-divergence but for the symmetric difference hypothesis class. A full description of these terms can be found in the supplementary material.
sections.heading: 3 Experiments
sections.text: In this section we describe our methodology and experimental setup for addressing the problem of implicit domain generalization of ERMs. We first start with a brief overview of the DomainBed benchmark and then describe the collection of models we train in an effort to measure generalization.
sections.heading: 3.1 DomainBed Benchmark and Training Setup
sections.text: DomainBed (Gulrajani & Lopez-Paz, 2020)1 is a recently released toolbox for domain generalization which provides implementations of various domain generalization algorithms such as Invariant Risk Minimization (Arjovsky et al., 2019), Variance Risk Extrapolation (Krueger et al., 2020), and metalearning based domain generalization (Li et al., 2017a) in addition to unsupervised domain adaptation approaches (Sun & Saenko, 2016; Ganin et al., 2015; Yan et al., 2020). It also provides various datasets such as RotatedMNIST (Ghifary et al., 2015), VLCS (Fang et al., 2013), and PACS (Li et al., 2017b). These datasets are provided in the form of multiple environments e = {(Xie, yie)}Ni=1, each
  1https://github.com/facebookresearch/DomainBed (MIT License)
  with a different training distribution pe(X) and (generally) the same conditional label distribution p(y∣X). Domain generalization algorithms are then trained on a subset of source environments Esource = {ejsource}Mj=1 and tested on target environments Etarget = {e j target}Kj=1.
  An intriguing result from the DomainBed benchmark is that standard ERM training can match or outperform more specialized domain generalization algorithms. We would like to understand this phenomenon better by characterizing properties that distinguish ERMs (Empirical Risk Minimizers) which perform well from those which do not. We follow Gulrajani & Lopez-Paz (2020) in combining all the training environments into one source dataset S = ⋃e∈Etrain e used to train our ERMs. Testing is performed independently on each target environment.
  We use the same network architectures as those used in the DomainBed paper (see supplemental information for more details). Overall our networks span shallow convolutional neural networks with 386K parameters (on the MNIST datasets) to residual networks with 25 million parameters (on VLCS and PACS). We prioritize a careful, exhaustive hyperparameter sweep with the above architectures, to obtain models with different levels of performance (Gulrajani & Lopez-Paz, 2020), over studying different architectures which is arguably a second order effect on this problem. Following Gulrajani & Lopez-Paz (2020), for each combination of dataset and training environments, we pick 100 random hyperparameter settings of batch size, learning rate, weight decay, and dropout (for resnet models). For both source S and target T , we hold out 50% of the data for validation. Since no training happens on T this means that we can use 50% of the data in T for computing generalization measures and the other 50% for assessing the out-of-distribution error of an an empirical risk minimizer ĥ, which we term as T (ĥ). Analogously, we denote the validation error of the model on the source domain as S(ĥ).
sections.heading: 3.2 Trained Empirical Risk Minimizers (ERMs) and Implicit Domain Generalization
sections.text: Using the above procedure, we train approximately 12,000 models on a compute cluster with Volta GPUs using PyTorch (Paszke et al., 2019). All models are trained for 5000 training steps and the model saved at the last step is used for analysis (Gulrajani & Lopez-Paz, 2020). fig. 2 shows the results of the sweep. We note that the source-domain validation accuracy or error S(ĉ) is a good candidate for explaining performance on domain generalization (see fig. 2). This is a common practice in the literature (Hendrycks et al., 2020), where model selection is often performed based on this criteria. However, we note that for a given source-domain validation accuracy, there is significant variation in the corresponding out-of-distribution accuracy, suggesting need for measures to characterize generalization more tightly.
sections.heading: 4 Testing the Predictive Power of Theory-based Measures
sections.text: In this section, we empirically evaluate various elements of the theory discussed in section 2 on 12,000 neural networks trained via ERM on the DomainBed benchmark.
  Our general methodology for testing the theory involves two major components. Firstly, we estimate the tightness in the bounds from Ben-David et al. (2007) empirically, and secondly, we study if the terms from the theory are predictive of performance. As explained in section 2, given D̃S , the set of features extracted on a source domain, and D̃T , the set of features extracted on the target domain, the
  theory provides bounds on the target error T (c) in terms of the source error S(c), H-divergence (dH(D̃S , D̃T )) and closeness of the domains (λ) (both computed in the feature space of a model):
  T (c) ≤ S(c) + dH(D̃S , D̃T ) + λ, (1)
  Estimation Considerations: In order to test how well the theory works in practice, we need to estimate the H and the H∆H-divergences from finite samples. We construct and empirically study the variance of estimators for both in the supplementary material, finding that estimates of H∆Hdivergence have a lower SNR than H-divergence (for which there are known convergence guarantees when H has a finite VC-dimension). However, for completeness we report results with both. How loose are the bounds? In fig. 3 we plot for each divergence measure and dataset, the right hand side of the bound in eq. (1). This reveals several interesting observations. Firstly, when the bounds, which are of the form “divergence measure + λ + S(ĉ)” (see eq. (1) for an example) are loose (first box-plot in each figure) (where looseness is defined as the difference between right hand side and left hand side of eq. (1)) it is not because the representations R entailed by the networks do not yield domains which are λ close but because the divergence between the source and target domains is large. Thus, in principle ERMs seem to yield feature spaces from which one can support the learning of both target and source domain labels. However, ERMs do not appear to learn feature spaces which implicitly align the source domain to the (unseen) target domain in the domain generalization setting. In unsupervised domain adaptation (uDA) one could enforce the divergence to be small (Ganin et al., 2015) (since uDA assumes access to the target domain inputs D̃T ), which could result in tighter bounds on test error, as long as the domains remain lambda-close. However, recent work (Zhao et al., 2019; Wu et al., 2019) shows that enforcing a low divergence can yield representations which are no longer lambda close, rendering the theory less useful in practice from an uDA perspective. Thus, having a low divergence seems neither sufficient nor necessary for generalization.
  Our goal is not just to understand how tight or loose the bounds are, but to check how well the measures computed based on them correlate with target-domain test error. What governs the regimes in which the bounds correlate well with test error T (ĥ)? Do tighter bounds lead to stronger correlation with test error? We find that this is not necessarily the case, for example, on RotatedMNIST with H∆H-divergence-MS based bound (H∆H-divergence-MS + λ + S(ĉ)) (where the gap is very low (fig. 3, right, yellow-boxes) the Spearman’s ρ with observed generalization T (ĥ) is 0.422 while on VLCS it is 0.567. Thus, it is not just how biased upper bound is, but also the variance which can play a major role in predicting generalization.
  Why do variations in H-divergence not correlate well with observed generalization despite λcloseness? We illustrate with a simple example (fig. 4) how one can obtain good generalization despite being very far from the source domain in terms of H-divergence. In such a regime, one can obtain a high H-divergence even when the classifier learnt on the source domain generalizes well to the target domain. Of course, the theory accounts for this in principle, since the bound on the target error is not violated. However, it is not useful in practice for predicting generalization despite the λ-closeness assumption being satisfied.
  Is the theory useful for predicting generalization? In order to use the bounds in a more practical setting, we must recall that we do not have access to the test labels from the target domain, and thus cannot compute λ-closeness. Instead, we consider a biased estimate of the bound, namely S(ĥ) +
  H-divergence, for the H-divergence based measures as an illustration. Note that one can understand λ-closeness as an assumption in the original theory, as opposed to something that is intended to be empirically computed. Thus, in this section we compute practically accessible components of the theory and measure correlation with actual error. Overall, we find that the Spearman’s ρ across conditions drops substantially without accounting for λ-closeness. For example, for H∆Hdivergence-MS based bound the Spearman’s ρ with generalization drops from 0.496 to 0.136 on removing λ-closeness. Similar trends can be seen for H∆H-divergence-SS on removing λ from the bound, which drops from 0.509 to 0.152 and H-divergence which goes from 0.215 to 0.027. In a sense, this is not very surprising, since λ-closeness uses the target-domain labels which provides significant signal for predicting the target domain error. However, this indicates that the theory cannot be used to great effect for predicting generalization in practice. Finally notice that among these measures, one performs much better generally when using H∆H-divergence-MS or H∆H-divergence-SS than when using H-divergence.
sections.heading: 5 Exploring Empirical Measures
sections.text: We have seen that the classic theory of domain generalization does not succeed at providing measures that are strongly predictive of out-of-distribution generalization ( T (ĉ)) in practice. Thus we next turn to exploring new measures that may provide better predictive power. We emphasize that our goal here is not to formulate highly predictive measure for use in practice, but rather to point towards promising directions for future theoretical investigations.
  In pursuit of these goals, given the heterogeneity of the environments (see fig. 2), we make the following methodological choices: (1) we seek good predictors of generalization for each combination of dataset and environment (which we term a condition) and aggregate across conditions to find measures which are reliable across different conditions, (2) we learn both separate regressors for each condition and a single regressor across all conditions. In all cases, we perform Ridge regression with leave-one-out cross validation for all the analysis. Across all regressions, we use a train, test split of (0.7, 0.3). We use the coefficient of determination R2 as a metric which indicates the fraction of the variance of the target variable explained by the regression, and report the average R2 of a predictor across all conditions of dataset/environment pairs. Following the methodology of Jiang et al. (2019), we also compute and emphasize Spearman’s rank correlation coefficient (Spearman’s ρ) to quantify how well each measure predicts which ERMs will generalize out-of-distribution.
sections.heading: 5.1 Description of selected measures
sections.text: Here we describe select measures that quantify the degree to which the model is sensitive to changes in the distribution of inputs. Here we briefly summarize a few of the most performant measures we find. A full list can be found in the supplemental information, including variants of many measures. First, we briefly introduce measures based on the Fisher information of the model and how it changes
  when computed on samples from the source or target environment. Then we study the norm of the Jacobian of the model computed on source or target environment samples. Finally, we introduce a measure based on the entropy of the model’s predictions on source and target samples.
  Fisher Based Measures. The Fisher information (Amari, 1998) measures the sensitivity of model outputs to changes in the parameters. We expect that models will generalize well out-of-distribution when the parameter directions that affect model outputs are similar when computed on train and test input samples. Let the model output for class i on an input sample xn be hθ,i(xn), where θ denotes the model parameters. The Fisher information is then defined as I = 1
  N A T A, where
  Aµ,(in) = 1√ hθ,i(xn) ∂hθ,i(xn) dθµ . Our generalization measures based on the Fisher information compare the Fisher obtained on the source training dataset IStr to the Fisher obtained on the target data ITtr . Note that the Fisher, unlike the Hessian does not require labeled data and thus can be computed on the unlabeled images. For details, see the supplementary material.
  Jacobian Norm Based Measures. The input-output Jacobian, a measure of sensitivity of the model output to changes in the input, is defined as Ji,α =
  ∂hi(x) ∂xα
  . Intuitively, a model that is less sensitive to changes in the input will generalize better. Thus, it has been studied in the context of robust learning (Hoffman et al., 2019) and in-distribution generalization (Novak et al., 2018), where it was found to be predictive of generalization at the level of individual test points. Similar to Novak et al. (2018), we compute the Frobenius norm of the Jacobian. See supplementary material for full details.
  Mixup Based Measures. Mixup (Zhang et al., 2018) was proposed as a more robust alternative to ERM that encourages the learned function to be smooth, and has been shown to smooth the network’s input-output Jacobian (Carratino et al., 2020). Here we use mixup as a generalization measure, and use the model’s score function h instead of the labels y for interpolation. Our motivation is the same as that for the Jacobian: if the function is less smooth around target examples, the network should not generalize as well. Given a neural network h(X), a dataset T , and λ ∼ Beta(α, α), we compute the Mixup measure as: 1∣T ∣ ∑Xi∈T ,Xj∈T ;i≠j (λh(Xi) + (1 − λ)h(Xj) − h(λXi + (1 − λ)Xj))
  2. See supplemental material for a detailed algorithmic description for computing this measure.
  Entropy on Target Data. We also compute the output-entropy of the neural network h on the target domain data Ttr (Entropy-Target): 1∣Ttr∣ ∑Xi∈Ttr ∑ Y j=1 − log (h(Xi)[j]) ⋅ h(Xi)[j]. Entropy-Source is defined analogously but on source domain samples.
sections.heading: 5.2 Results with Single Measures
sections.text: We provide a representative sampling of the results for the measures we study in this section, while deferring to the appendix for more detailed results spanning all the measures that we study. For this section, we first computed the Spearman’s ρ for each of the measures with target-domain error, T (ĉ), and selected the measure from each family of related measures (say, those related to the Jacobian for example) which has the highest correlation. As a pre-processing step, we canonicalize the measures to always have a positive correlation with T (ĉ) (see appendix for more details). Which measures outperform the theory based measures H∆H-divergence-SS and Hdivergence? In terms of Spearman’s ρ, we find that Entropy-Source (0.598), Entropy-Target (0.734), Jacobian (0.420), MMD-Gaussian (0.283), and Fisher Align (0.441) all outperform the theory inspired measures from section 4, where the best performing measure was H∆H-divergence-SS at Spearman’s ρ of 0.152 table 1. Intriguingly, Entropy-Target (0.734) outperforms in-domain test error, S(ĉ), (0.712) in terms of Spearman’s ρ, while using R2 it is slightly worse (0.608 v.s. 0.582). Practically analyzing the performance of Entropy-Target. We next analyze the performance of Entropy-Target across the different datasets. We find that Entropy-Target outperforms S(ĉ) on the RotatedMNIST dataset (0.853 v.s. 0.744 Spearman’s ρ), while on the VLCS and PACS datasets S(ĉ) is better, with 0.625 v.s. 0.557 Spearman’s ρ on VLCS and 0.732 v.s. 0.662 Spearman’s ρ on PACS. One plausible explanation for the strong performance in this regime is shown in fig. 5. Consider the case where we train with environments with rotation 30-75 degrees in RotatedMNIST (as in fig. 1). In such a situation S(ĉ) cannot distinguish between the expected performance on the 0 degree and 15 degree test environments.
  Table 1: Different measures (rows) evaluated on either Spearman’s ρ or R2 metric between the measure and T (ĉ) (columns) where each measure is either used Alone as a predictor or in conjunction with in-domain test error, S(ĉ), and is allowed to use only one set of parameters across all datasets and source environments (Joint) or allowed to set parameters differently for different environments (Stratified). We report per-environment correlation averaged across environments. Approaches which perform better than source-domain validation accuracy are shown in bold, while approaches
  which perform better than the theory based measures are highlighted (p-values for all the results below are orders of magnitude lower than performance numbers, and are thus dropped).
  However we see in fig. 5 that the entropy distributions across different models computed on the 0 and 15 degree target environments are quite different, with higher confidence on the 15 degree test environment where better generalization occurs.
  What contributes to the performance of Entropy-Source? We find that across all models, the Entropy-Source (computed on the source training set) is highly correlated with S(ĉ) (e.g. Spearman’s ρ of 0.857 on RotatedMNIST). This suggests that the performance of Entropy-Source is likely because of its relationship to S(ĉ). This also potentially explains why using both S(ĉ) and Entropy-Source jointly table 1 has the same performance as using S(ĉ) alone (0.712 Spearman’s ρ).
  Performance of measures split across datasets. In fig. 6, we find that for many measures the correlation has the same sign across the different datasets, which is encouraging and indicates that the proposed measures might be useful to understand generalization. Overall we find that EntropySource, Entropy-Target, H-divergence (train), Fisher Align, Mixup, L2-Path Norm, MMD-Gaussian are reliable in terms of the sign of the correlation. However, H-divergence (train) and L2-Path Norm have a high variance across datasets, which is not ideal.
  Do Jacobian or Fisher measures perform better? While both Jacobian and Fisher Align have similar Spearman’s ρ values overall (around ∼ 0.4) we find that the Jacobian, while highly performant
  on RotatedMNIST with Spearman’s ρ of 0.83 across environments, performs quite poorly on PACS and VLCS (Spearman’s ρ ∼ 0), and thus may not be a reliable measure. Previous work has utilized the Jacobian (Novak et al., 2018) on four different image datasets (CIFAR10, FashionMNIST, MNIST, CIFAR100) and found it to be predictive of generalization at the level of individual test points coming from the same distribution. However, our results illustrate that in the out-of-distribution setting, the Jacobian is less reliable. In contrast, the measure Fisher Align is more stable across datasets (Spearman’s ρ of 0.368 on VLCS, 0.212 on PACS and 0.585 on RotatedMNIST). Our intuition for Fisher Align is that since the eigenvectors of the Fisher indicate the directions in parameter space in which the model is most sensitive, if these directions match on a target distribution, one is more likely to generalize.
sections.heading: 5.3 Results with Compound Measures
sections.text: In the previous sub-section we studied each of the proposed measures in isolation. Next, we study if combining the measures with in-domain test error, S(ĉ), can yield better performance. This is similar to the theory, which bounds the gap between target and source domain validation errors. In each case, we fit a regression model of the form α S(ĉ) + β ∗ feature to predict T (ĉ) and then use the predictions to compute Spearman’s ρ. However, we also report R2 of the regression in table 1.
  Joint model across all environments. We first perform the regression (as explained in section 3) across all datasets and environments jointly (table 1, “Joint” column). This does not allow the model to incorporate the specific environment in the weights of the regressor, but instead insists on the same predictor or “rule” across all environments. This allows a direct comparison to the experiments we discussed in the previous sub-section. We find that in this setup, only Entropy-Target, Jacobian and MMD-Mean-Cov when used in conjunction with S(ĉ) improve Spearman’s ρ (table 1), indicating that these measures might be complementary to S(ĉ). Stratified models for each environment. We next allow the regressor to choose a new set of parameters for each environment. In such a setting, it is impossible to perform worse than S(ĉ), since one can always recover a solution which places 0 weight on the second feature if that is indeed the best solution to be found. As expected, table 1 reveals that all measures improve in conjunction with S(ĉ) in this setting. However, it is still instructive to study which measures improve the performance of S(ĉ) the most, relative to others. We find that MMD-Gaussian, MMD-Mean-Cov, Jacobian, Entropy-Target all lead to Spearman’s ρ values greater than 0.80 in this setting, suggesting them to be promising candidates to pursue further in this context.
  Which is better, L2-Path Norm or Mixup? While L2-Path Norm appears to have a higher Spearman’s ρ than Mixup in the Alone setting (table 1), it appears that most of the benefit from L2-Path Norm comes from it being a good predictor of in-distribution generalization (Neyshabur et al., 2015). When paired with S(ĉ), even in the stratified setting, we find that L2-Path Norm has the same performance as S(ĉ) (0.712 Spearman’s ρ), whereas Mixup improves S(ĉ) from 0.712 to 0.749 Spearman’s ρ. Similar trends can be observed for R2 as well.
sections.heading: 6 Related work and Limitations
sections.text: Our work first sets out to test the seminal theory around unsupervised domain adaptation (BenDavid et al., 2007, 2010), which aims to understand when an approach is expected to generalize to test domains different from the training domain. We provide an empirical evaluation of some of the bounds and theory, and surprisingly find that less principled approaches based on the Fisher information and Jacobian Norm perform better than bounds inspired by domain adaptation theory.
  Most related to our current work is the work on DomainBed (Gulrajani & Lopez-Paz, 2020) which observed that more careful experimental choices lead to ERMs that are competitive with state of the art domain generalization methods. Our work seeks to understand what characterizes ERMs that generalize, with our results suggesting the entropy of the output distribution, norm of the Jacobian, and measures based on the Fisher being promising candidates for future research.
  Other related work (Zhao et al., 2019; Wu et al., 2019) finds that the theory of unsupervised domain adaptation (Ben-David et al., 2007, 2010) when used to minimize domain divergences (Ganin et al., 2015) is not always useful in practice, in terms of reducing target domain error. Our work is different in that we study domain generalization (as opposed to adaptation) where one cannot minimize the
  domain divergences explicitly. Correspondingly, we uncover that the theory is not useful in practice for predicting generalization as opposed to enforcing it.
  Our methodological approach also takes inspiration from Jiang et al. (2019); Neyshabur et al. (2017) which aims to understand how overparameterized deep networks are able to achieve good generalization, motivated in part by their ability to fit random labels (Zhang et al., 2016). We implement some of the best performing measures from the large-scale empirical study of Jiang et al. (2019) for the in-domain setting, and employ rank correlation as a measure of performance following this work. We believe it is a fruitful intellectual and practical challenge to understand the role of implicit regularization (Neyshabur et al., 2015) in the domain generalization setting as well.
  Previous work (Hoffman et al., 2019; Drucker & Le Cun, 1992; Varga et al., 2017) has studied domain generalization using the Jacobian norm as a regularizer. While their aim is to build a more robust model for supervised learning, our goal is in characterizing if an ERM trained without any regularization can be shown to be implicitly learning smooth functions, and therefore help aid understanding of the implicit domain generalization phenomenon. Novak et al. (2018) show that the Jacobian norm is useful for predicting generalization at the level of individual test points for in-domain classification. Here we extend their experimental setting to the out-of-distribution case, and show that the Jacobian is useful at the population level. Jastrzebski et al. (2020) recently studied the role of the evolution of the trace of the Fisher information over the course of training and its impact on in-domain generalization and proposed a regularizer based on this insight. We utilize the Fisher for the domain generalization setting. In contrast to Jastrzebski et al. (2020) who study the trace of the Fisher, our best Fisher-based method uses the eigenvectors for training and test.
  Finally, predicting out-of-distribution generalization is related to out-of-distribution detection, but there is an important difference. While in out-of-distribution detection the task is to flag out-ofdistribution examples that do not belong to any of the classes observed during training (Hendrycks et al., 2018), in out-of-distribution generalization we assume that test instances always belong to one of the classes of interest, albeit they are generated from a different distribution during test time.
  Limitations: Our work is a step towards building an understanding of when networks trained with ERM generalize to novel target domains. However, our study is empirical, which limits the generality of our conclusions. We hope this effort spurs future work on theoretical aspects of this problem. While we follow the model architectures of Gulrajani & Lopez-Paz (2020) to obtain a large set of pretrained models, studying the impact of architectures in addition to hyperparameters is also an interesting line of study.
sections.heading: 7 Conclusion
sections.text: In this work, we studied the domain generalization performance of ERM trained deep neural networks. Surprisingly, we found that adapting theory from unsupervised domain adaptation was of limited use in predicting how well models generalize out-of-distribution. We found that the assumptions of the theory were satisfied, in particular λ-closeness, yet the practically accessible components of the theory do not provide strong predictions about which models will generalize. In an effort to discover other correlates of domain generalization, we studied the performance of a suite of novel empirical measures, finding a number of promising candidates. Our work provides a critical first step for understanding the implicit domain generalization properties of ERMs and points towards promising future theoretical investigations into strongly predictive measures of domain generalization.
  Social Impact
  While most work that improves the performance of machine learning models can be leveraged by applications that have negative societal impact, we believe that the results of this paper actually take steps in a positive direction for the deployment of machine learning models in broader society. In particular, our work aims to understand when models that were not trained to be robust to domain shift break down or actually succeed on novel test domains. A better understanding and ability to predict model behavior in such situations should help in building more trustworthy machine learning methods that are less likely to result in unintended consequences when deployed on novel data. We emphasize that our work does not propose a new algorithm for domain generalization but rather attempts to predict when previously trained models can be trusted to perform in novel domains.
sections.heading: Acknowledgments
sections.text: We would like to thank Kamalika Chaudhuri, Pascal Vincent, Ricardo Monti, Edward Grefenstette, Yann Dubois and numerous other colleagues at Facebook for discussions and feedback on the paper. We also thank Kamalika for detailed feedback on a draft of the paper.
  Funding Transparency Statement
  D.J.S was partially supported by the NIH under award R01EB026943, the NSF through the Center for the Physics of Biological Function (PHY-1734030), and as a Simons Investigator in the Mathematical Modeling of Living Systems (MMLS).
title: An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers