summaries: The paper attempts to provide a solution to an interesting problem of cross language retrieval based question answering in a zero shot (without requirement of any language-specific annotated training data). Authors propose a new DPR algorithm that to be able to retrieve documents relevant to the question across languages (for multiple languages). The paper extends retrieve-the-generate based open domain QA system to a cross-lingual setting.

summaries: This paper addresses the multilingual open QA task by combining a cross-lingual passage retrieval algorithm with a multilingual autoregressive generation model. The approach is thus capable of answering questions where the answer and the question are in different languages.  The authors extended existing annotated data using an iterative training approach that improves retrieval for languages with limited resources. The experimental results showed that the proposed method (CORA) outperformed the previous state of the art on multilingual open question answering benchmarks across 26 languages.

summaries: This paper studies the task of multilingual Open-Domain Question Answering.  The task setting requires building a system that can answer natural language questions in any language on an open set of domains. Models are required to answer in the same language the question was asked in.  The authors tackle the task using a fairly conventional retrieve-and-read ODQA model, with the necessary modifications to make both the retriever and reader component language-agnostic, by exploiting multi-lingual pretrained models, training on available multilingual training data, and using and a clever data augmentation setup to increase multilingual training data.   The proposed solution “CORA”, operates over wikipedias in many different languages.   The retrieval component “MDPR” is a multi-lingual extension of DPR, using MBERT. This model can retrieve evidence documents across different languages, rather than having to match the language of the question. This means that a question expressed in a low-resource language can benefit from retrieving evidence from high resource languages. It also allows for a question written in one language on a cultural topic associated with a different language to retrieve from the most appropriate language’s wikipedia.   The reader “mGEN” is a generative reader, implemented using mT5, that takes as input the concatenation of the top-K retrieved passages from the retriever (which may be several different languages) and the question, and generates the answer in the question language, token by token.   The main difficulty of the proposed solution is how to train the MDPR and MGEN model components, given the limited multilingual raining data. The initial model is trained with NQ (english) and TyDi QA and Xor-TiDi QA (multilingual data). To increase the amount of multilingual training data and language coverage,  the authors introduce a data augmentation strategy where the initial model is used to harvest potential additional data. This exploits the wikipedia “language links” resource. This iterative approach works by retrieving passages, looking up passages on the same topic in other languages using “language links”, and using the reader to see whether these can be used to generate the correct answer - if they can, they are added as additional training data for the next round. Another expansion mechanism exploits the fact that most answers are wikipedia entities, and can be translated using annotations available in the wikipedia “language links” resource.  CORA is thoroughly evaluated on TyDI QA and an evaluation-only dataset called MKQA, and performs very strongly, improving the state-of-the-art by a wide margin in almost every setting.. Thorough ablations give insights and quantify the the effects of the system, and demonstrate zero-shot question answering abilities

summaries: This paper presents a multilingual dense passage retrieval model called mDPR to deal with the multilingual/cross-lingual IR and QA tasks. The mDPR part is firstly trained using existing labeled multilingual Q-P pairs. Then, possible passage pairs are selected from a given multilingual corpus based on the trained mDPR and used to train an answer generation model called mGEN. Next, given each question, the trained mDPR is further used to retrieve more possible passage candidates in other languages from Wikipedia corpus, where the retrieved passages that can lead to correct answers are considered as positive instances, the others as negative instances. With these expanded new question-passage pairs, the training of mDPR is performed again. The above procedure is conducted in an iterative manner. Evaluations are conducted on TyDi QA and MKQA in the open-domain QA setting. Several baselines are included and compared. The key contribution is to do DPR in the multilingual/cross-lingual setting.

abstractText: We present Cross-lingual Open-Retrieval Answer Generation (CORA), the first unified many-to-many question answering (QA) model that can answer questions across many languages, even for ones without language-specific annotated data or knowledge sources. We introduce a new dense passage retrieval algorithm that is trained to retrieve documents across languages for a question. Combined with a multilingual autoregressive generation model, CORA answers directly in the target language without any translation or in-language retrieval modules as used in prior work. We propose an iterative training method that automatically extends annotated data available only in high-resource languages to low-resource ones. Our results show that CORA substantially outperforms the previous state of the art on multilingual open QA benchmarks across 26 languages, 9 of which are unseen during training. Our analyses show the significance of cross-lingual retrieval and generation in many languages, particularly under low-resource settings. Our code and trained model are publicly available at https://github.com/AkariAsai/ CORA.

authors:
Chris Alberti
Daniel Andor
Emily Pitler
Jacob Devlin
Michael Collins.

title: Synthetic QA corpora generation with roundtrip consistency

venue: ACL.

authors:
Akari Asai
Eunsol Choi.

title: Challenges in information seeking QA: Unanswerable questions and paragraph retrieval

authors:
Akari Asai
Jungo Kasai
Jonathan H. Clark
Kenton Lee
Eunsol Choi
Hannaneh Hajishirzi.

title: XOR QA: Cross-lingual open-retrieval question answering

venue: NAACL.

authors:
Jan A. Botha
Zifei Shan
Daniel Gillick.

title: Entity Linking in 100 Languages

venue: EMNLP.

authors:
Danqi Chen
Adam Fisch
Jason Weston
Antoine Bordes.

title: Reading Wikipedia to answer open-domain questions

authors:
Jonathan H. Clark
Eunsol Choi
Michael Collins
Dan Garrette
Tom Kwiatkowski
Vitaly Nikolaev
Jennimaria Palomaki.

title: TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages

venue: TACL.

authors:
Alexis Conneau
Kartikay Khandelwal
Naman Goyal
Vishrav Chaudhary
Guillaume Wenzek
Francisco Guzmán
Edouard Grave
Myle Ott
Luke Zettlemoyer
Veselin Stoyanov.

title: Unsupervised cross-lingual representation learning at scale

authors:
Nicola De Cao
Ledell Wu
Kashyap Popat
Mikel Artetxe
Naman Goyal
Mikhail Plekhanov
Luke Zettlemoyer
Nicola Cancedda
Sebastian Riedel
Fabio Petroni

title: Multilingual autoregressive entity linking

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova.

title: BERT: Pre-training of deep bidirectional transformers for language understanding

authors:
Hao Fei
Meishan Zhang
Donghong Ji.

title: Cross-lingual semantic role labeling with highquality translated training corpus

authors:
Daniel Gillick
Sayali Kulkarni
Larry Lansing
Alessandro Presta
Jason Baldridge
Eugene Ie
Diego Garcia-Olano.

title: Learning dense representations for entity retrieval

venue: CoNLL.

authors:
Kelvin Guu
Kenton Lee
Zora Tung
Panupong Pasupat
Ming-Wei Chang.

title: Realm: Retrieval-augmented language model pre-training

venue: ICML.

authors:
Gautier Izacard
Edouard Grave.

title: Distilling knowledge from reader to retriever for question answering

venue: ICLR.

title: Leveraging passage retrieval with generative models for open domain question answering

venue: EACL.

authors:
Marcin Junczys-Dowmunt
Roman Grundkiewicz
Tomasz Dwojak
Hieu Hoang
Kenneth Heafield
Tom Neckermann
Frank Seide
Ulrich Germann
Alham Fikri Aji
Nikolay Bogoychev
André F.T. Martins
Alexandra Birch.

title: Marian: Fast neural machine translation in C++

venue: ACL (System Demonstrations).

authors:
Vladimir Karpukhin
Barlas Oğuz
Sewon Min
Ledell Wu
Sergey Edunov
Danqi Chen
Wen-tau Yih.

title: Dense passage retrieval for open-domain question answering

authors:
Diederik P Kingma
Jimmy Ba.

title: Adam: A method for stochastic optimization

authors:
Dan Kondratyuk
Milan Straka.

title: 75 languages, 1 model: Parsing Universal Dependencies universally

authors:
Tom Kwiatkowski
Jennimaria Palomaki
Olivia Redfield
Michael Collins
Ankur Parikh
Chris Alberti
Danielle Epstein
Illia Polosukhin
Jacob Devlin
Kenton Lee
Kristina Toutanova
Llion Jones
Matthew Kelcey
Ming-Wei Chang
Andrew M. Dai
Jakob Uszkoreit
Quoc Le
Slav Petrov.

title: Natural Questions: A benchmark for question answering research

authors:
Kenton Lee
Ming-Wei Chang
Kristina Toutanova.

title: Latent retrieval for weakly supervised open domain question answering

authors:
Patrick Lewis
Ethan Perez
Aleksandara Piktus
Fabio Petroni
Vladimir Karpukhin
Naman Goyal
Heinrich Küttler
Mike Lewis
Wen-tau Yih
Tim Rocktäschel

title: Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS

authors:
Jimmy Lin
Xueguang Ma
Sheng-Chieh Lin
Jheng-Hong Yang
Ronak Pradeep
Rodrigo Nogueira

title: Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations

authors:
Jimmy Lin
Rodrigo Nogueira
Andrew Yates

title: Pretrained transformers for text ranking: BERT and beyond

authors:
Yankai Lin
Haozhe Ji
Zhiyuan Liu
Maosong Sun.

title: Denoising distantly supervised open-domain question answering

authors:
Yinhan Liu
Jiatao Gu
Naman Goyal
Xian Li
Sergey Edunov
Marjan Ghazvininejad
Mike Lewis
Luke Zettlemoyer.

title: Multilingual denoising pre-training for neural machine translation

authors:
Shayne Longpre
Yi Lu
Joachim Daiber

title: MKQA: A linguistically diverse benchmark for multilingual open domain question answering

authors:
Chunchuan Lyu
Shay B. Cohen
Ivan Titov.

title: Semantic role labeling with iterative structure refinement

authors:
Sewon Min
Jordan Boyd-Graber
Chris Alberti
Danqi Chen
Eunsol Choi
Michael Collins
Kelvin Guu
Hannaneh Hajishirzi
Kenton Lee
Jennimaria Palomaki
Colin Raffel
Adam Roberts
Tom et al Kwiatkowski.

title: NeurIPS 2020 EfficientQA competition: Systems, analyses and lessons learned

venue: PMLR.

authors:
Sewon Min
Danqi Chen
Hannaneh Hajishirzi
Luke Zettlemoyer.

title: A discrete hard em approach for weakly supervised question answering

authors:
Phoebe Mulcaire
Jungo Kasai
Noah A. Smith.

title: Low-resource parsing with crosslingual contextualized representations

title: Polyglot contextual representations improve crosslingual transfer

authors:
Arij Riabi
Thomas Scialom
Rachel Keraron
Benoît Sagot
Djamé Seddah
Jacopo Staiano.

title: Synthetic data augmentation for zero-shot cross-lingual question answering

authors:
Adam Roberts
Colin Raffel
Noam Shazeer

title: How much knowledge can you pack into the parameters of a language model

authors:
Siamak Shakeri
Noah Constant
Mihir Sanjay Kale
Linting Xue

title: Towards zero-shot multilingual synthetic question and answer generation for cross-lingual reading comprehension

authors:
Saku Sugawara
Kentaro Inui
Satoshi Sekine
Akiko Aizawa

title: What makes reading comprehension questions easier

title: Parallel data, tools and interfaces in OPUS

venue: LREC.

authors:
Jörg Tiedemann
Lars Nygaard.

title: The OPUS corpus - parallel and free

authors:
Jörg Tiedemann
Santhosh Thottingal.

title: OPUS-MT — Building open translation services for the World

venue: EAMT.

authors:
Chau Tran
Yuqing Tang
Xian Li
Jiatao Gu.

title: Cross-lingual retrieval for iterative selfsupervised training

venue: NeurIPS.

authors:
Ferhan Ture
Elizabeth Boschee.

title: Learning to translate for multilingual question answering

authors:
Thomas Wolf
Lysandre Debut
Victor Sanh
Julien Chaumond
Clement Delangue
Anthony Moi
Pierric Cistac
Tim Rault
Remi Louf
Morgan Funtowicz
Joe Davison
Sam Shleifer
Patrick von Platen
Clara Ma
Yacine Jernite
Julien Plu
Canwen Xu
Teven Le Scao
Sylvain Gugger
Mariama Drame
Quentin Lhoest
Alexander Rush

title: Transformers: State-of-the-art natural language processing

venue: In EMNLP (System Demonstrations)

authors:
Lee Xiong
Chenyan Xiong
Ye Li
Kwok-Fung Tang
Jialin Liu
Paul N. Bennett
Junaid Ahmed
Arnold Overwijk.

title: Approximate nearest neighbor negative contrastive learning for dense text retrieval

authors:
Linting Xue
Noah Constant
Adam Roberts
Mihir Kale
Rami Al-Rfou
Aditya Siddhant
Aditya Barua
Colin Raffel.

title: mT5: A massively multilingual pre-trained text-to-text transformer

heading: 1 Introduction

text:
Multilingual open question answering (QA) is the task of answering a question from a large collection of multilingual documents. Most recent progress in open QA is made for English by building a pipeline based on a dense passage retriever trained on large-scale English QA datasets to find evidence passages in English (Lee et al., 2019; Karpukhin et al., 2020), followed by a reader that extracts an answer from retrieved passages. However, extending this approach to multilingual open QA poses new challenges. Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures (Asai et al., 2021; Lin et al., 2020). Nonetheless, large-scale cross-lingual open QA training data whose questions and evidence are in different languages are not available in many of those languages.
To address these challenges, previous work in multilingual open QA (Ture and Boschee, 2016; Asai et al., 2021) translates questions into English, applies an English open QA system to answer in English, and then translates answers back to the target language. Those pipeline approaches suffer from error propagation of the machine translation component into the downstream QA, especially for low-resource languages. Moreover, they are not able to answer questions whose answers can be found in resources written in languages other than English or the target languages.
In this paper, we introduce a unified many-to-many QA model that can answer questions in any target language by retrieving evidence from any language and generating answers in the target language. Our method (called CORA, Fig. 1) extends the retrieve-then-generate approach of English open QA (Lewis et al., 2020; Izacard and Grave, 2021b) with a single cross-lingual retriever and a generator that do not rely on language-specific retrievers or machine translation modules. The multilingual
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
retrieval module (mDPR) produces dense embeddings of a question and all multilingual passages, thereby retrieving passages across languages. The generation module (mGEN) is trained to output an answer in the target language conditioned on the retrieved multilingual passages. To overcome the aforementioned data scarcity issue, we automatically mine training data using external language links and train mDPR and mGEN iteratively. In particular, each iteration proceeds over two stages of updating model parameters with available training data and mining new training data cross-lingually by Wikipedia language links and predictions made by the models. This approach does not require any additional human annotations or machine translation, and can be applied to many new languages with low resources.
Our experiments show that CORA advances the state of the art on two multilingual open QA datasets, XOR-TYDI QA (Asai et al., 2021) and MKQA (Longpre et al., 2020), across 26 typologically diverse languages; CORA achieves gains of 23.4 and 4.7 F1 points in XOR-TYDI QA and MKQA respectively, where MKQA data is not used for training. Moreover, CORA achieves F1 scores of roughly 30 over 8 languages on MKQA that have no training data or even reference Wikipedia documents, outperforming the state-of-the-art approach by 5.4 F1 points. Our controlled experiments and human analyses illustrate the impact of many-to-many cross-lingual retrieval in improving multilingual open QA performance. We further observe that through cross-lingual retrieval, CORA can find answers to 20% of the multilingual questions that are valid but are originally annotated as unanswerable by humans due to the lack of evidence in the English knowledge sources.

heading: 2 Method

text: We define multilingual open QA as the task of answering a question qL in a target language L given a collection of multilingual reference passages Cmulti, where evidence passages can be retrieved from any language. These passages come from Wikipedia articles that are not necessarily parallel over languages. We introduce CORA, which runs a retrieve-then-generate procedure to achieve this goal (Fig. 1). We further introduce a novel training scheme of iterative training with data mining (§ 2.2).

heading: 2.1 CORA Inference

text:
CORA directly retrieves evidence passages from any language for questions asked in any target language, and then generates answers in the target language conditioned on those passages. More formally, the CORA inference consists of two steps of (i) retrieving passages Pmulti and (ii) generating an answer aL based on the retrieved passages. Pmulti can be in any language included in Cmulti.
Pmulti = mDPR(qL,Cmulti), aL = mGEN(qL,Pmulti).
Multilingual Dense Passage Retriever (mDPR). mDPR extends Dense Passage Retriever (DPR; Karpukhin et al., 2020) to a multilingual setting. mDPR uses an iterative training approach to fine-tune a pre-trained multilingual language model (e.g., mBERT; Devlin et al., 2019) to encode passages and questions separately. Once training is done, the representations for all passages from C
multi are computed offline and stored locally. Formally, a passage encoding is obtained as follows: epL = mBERTp(p), where a passage p is a fixed-length sequence of tokens from multilingual documents. At inference, mDPR independently obtains a d-dimensional (d = 768) encoding of the
question eqL = mBERTq(qL). It retrieves k passages with the k highest relevance scores to the question, where the relevance score between a passage p and a question qL is estimated by the inner product of their encoding vectors, ⟨eqL , ep⟩. Multilingual Answer Generator (mGEN). We use a multilingual sequence-to-sequence model (e.g., mT5; Xue et al., 2021) to generate answers in the target language token-by-token given the retrieved multilingual passages Pmulti. We choose a generation approach because it can generate an answer in the target language L from passages across different languages.1 Moreover, the generator can be adapted to unseen languages, some of which may have little or no translation training data. Specifically, the generator outputs the sequence probability for aL as follows:
P (aL∣qL,Pmulti) = T
∏ i p(aLi ∣aL<i, qL,Pmulti), (1)
where aLi denotes the i-th token in the answer, and T is the length of the answer. We append a language tag to the question to indicate the target language.

heading: 2.2 CORA Training

text:
We introduce an iterative training approach that encourages cross-lingual retrieval and answer generation conditioned on multilingual passages (sketched in Fig. 2 and Alg. 1). Each iteration proceeds over two stages: parameter updates (§ 2.2.1) where mDPR and mGEN are trained on the current training data and cross-lingual data mining (§ 2.2.2) where training data are automatically expanded by Wikipedia language links and model predictions.
Initial training data. The initial training data is a combination of multilingual QA datasets: XORTYDI QA and TYDI QA (Clark et al., 2020), and an English open QA dataset (Natural Questions, Kwiatkowski et al., 2019). Each training instance from these datasets comprises a question, a positive passage, and an answer. Note that annotations in the existing QA datasets have critical limitations: positive passages are taken either from English (Asai et al., 2021) or the question’s language (Clark et al., 2020). Further, most of the non-English languages are not covered. Indeed, when we only train mDPR on this initial set, it often learns to retrieve passages in the same languages or similar languages with irrelevant context or context without sufficient evidence to answer.

heading: 2.2.1 Parameter Updates

text:
mDPR updates (line 3 in Alg. 1). Let D = {⟨qLi , p+i , p−i,1,⋯, p−i,n⟩}mi=1 be m training instances. Each instance consists of a question qLi , a passage that answers the question (positive passage) p + i , and n passages that do not answer the question (negative passages) p−i,j . For each question, we use positive passages for the other questions in the training batch as negative passages (in-batch negative, Gillick et al., 2019; Karpukhin et al., 2020). mDPR is updated by minimizing the negative log likelihood of positive passages:
Lmdpr = − log exp(⟨eqLi , ep+i ⟩)
exp(⟨eqLi , ep+i ⟩) +∑ n j=1 exp(⟨eqLi , ep−i,j ⟩)
. (2)
1An alternative approach of answer extraction requires translation for all language pairs (Asai et al., 2021).
Algorithm 1: Iterative training that automatically mines training data. Data: Input QA pairs: (qL, aL)
1 initialize training data B1 = (qL, aL, pgold),L = {Eng, L}; 2 while t < T do 3 Θ t mDPR ← Train(θt−1mDPR,Bt)/* Train mDPR */ 4 Pmulti ← mDPR(qL, embedding(Cmulti))/* Retrieve passages */ 5 θ t mGEN ← Train(θt−1mGEN , (qL, aL,Pmulti))/* Train mGEN */ 6 For L == Eng, Pmulti+ = LangLink(qL,Cmulti)) /* Mine data using Wikidata */ 7 For pi ∈ P
multi: if mGEN(qL, pi) == aL then positives.add(pi) else negatives.add(pi) 8 B
t+1 += (qL, aL, positives, negatives) /* Add new training data */ 9 t← t + 1
10 end
mGEN updates (lines 4-5 in Alg. 1). After updating mDPR, we use mDPR to retrieve top k passages Pmulti for each qL. Given these pairs of the question and the retrieved passages (qL,Pmulti) as input, mGEN is trained to generate answer aL autoregressively (Eq. (1)) and minimize the crossentropy loss. To train the model to generate in languages not covered by the original datasets, we translate aL to other languages using Wikipedia language links and create new synthetic answers.2 See Appendix § A.2 for more detail.

heading: 2.2.2 Cross-lingual Data Mining

text:
After the parameter updates, we mine new training data using mDPR and Wikipedia language links and label the new data by mGEN predictions. This step is skipped in the final iteration.
Mining by trained mDPR and language links (line 4, 6 in Alg. 1). Trained mDPR can discover positive passages in another language that is not covered by the initial training data. At each iteration, we use retrieved passages Pmulti for qL (line 4 in Alg. 1) as a source of new positive and negative passages. This enables expanding data between language pairs not in the original data.
To cover even more diverse languages, we use language links and find passages in other languages that potentially include sufficient evidence to answer. Wikipedia maintains article-level language links that connect articles on the same entity over languages. We use these links to expand training data from the English QA dataset of Natural Questions (line 6 in Alg. 1). Denote a training instance by (qEn, aEn, pgold). We first translate the English answer aEn to a target language aL using language links. We use language links again to look up the English Wikipedia article that the gold passage pgold comes from. We then find articles in non-English languages in the reference documents C multi that correspond to this article. Although the language link-based automatic translation cannot handle non-entity answers (e.g., short phrases), this helps us to scale to new languages without additional human annotation or machine translation. We add all passages from these articles to Pmulti as positive passage candidates, which are then passed to mGEN to evaluate whether each of them leads to aL or not.
Automatic labeling by mGEN predictions (lines 7-8 in Alg. 1). A passage pi from P multi may not always provide sufficient information to answer the question qL even when it includes the answer string aL. To filter out those spurious passages (Lin et al., 2018; Min et al., 2019), we take instances generated from the two mining methods described above, and run mGEN on each passage to predict an answer for the question. If the answer matches the correct answer aL, then the passage pi is labeled as a positive passage; otherwise we label the input passage as a negative passage. We assume that when mGEN fails to generate a correct answer given the passage, the passage may not provide sufficient evidence to answer; this helps us filter out spurious passages that accidentally contain an answer string yet do not provide any clue to answer. We add these new positive and negative passages to the training data, and in the next iteration, mDPR is trained on this expanded training set (§ 2.2.1).
2This automatic answer translation is only done after the third epoch of initial training to prevent the model from overfitting to synthetic data.

heading: 3 Experiments

text: We evaluate CORA on two multilingual open QA datasets across 28 typologically diverse languages.3 CORA achieves state-of-the-art performance across 26 languages, and greatly outperforms previous approaches that use language-specific components such as question or answer translation.

heading: 3.1 Datasets and Knowledge Sources

text:
Multilingual open QA datasets differ in covered languages, annotation schemes, and target application scenarios. We evaluate F1 and EM scores over the questions with answer annotations from two datasets, following the common evaluation practice in open QA (Lee et al., 2019).
XOR-TYDI QA. XOR-TYDI QA (Asai et al., 2021) is a multilingual open QA dataset consisting of 7 typologically diverse languages, where questions are originally from TYDI QA (Clark et al., 2020) and posed by information-seeking native speakers. The answers are annotated by extracting spans from Wikipedia in the same language as the question (in-language data) or by translating English spans extracted from English Wikipedia to the target language (cross-lingual data). XOR-TYDI QA offers both training and evaluation data.
MKQA. MKQA (Longpre et al., 2020) is an evaluation dataset created by translating 10k Natural Questions (Kwiatkowski et al., 2019) to 25 target languages. The parallel data enables us to compare the models’ performance across typologically diverse languages, in contrast to XOR-TYDI QA. MKQA has evaluation data only; XOR-TYDI QA and MKQA have five languages in common.
Collection of multilingual documents Cmulti. We use the February 2019 Wikipedia dumps of 13 diverse languages from all XOR-TYDI QA languages and a subset of MKQA languages.4 We choose 13 languages to cover languages with a large number of Wikipedia articles and a variety of both Latin and non-Latin scripts. We extract plain text from Wikipedia articles using wikiextractor,5 and split each article into 100-token segments as in DPR (Karpukhin et al., 2020). We filter out disambiguation pages that distinguish pages that share the same article title6 as well as pages with fewer than 20 tokens, resulting in 43.6M passages. See more details in Appendix § B.2.
Language categories. To better understand the model performance, we categorize the languages based on their availability during our training. We call the languages with human annotated gold paragraph and answer data seen languages. XOR-TYDI QA provides gold passages for 7 languages. For the languages in Cmulti without human-annotated passages, we mine new mDPR training data by our iterative approach (§ 2.2). We call these languages, which are seen during mDPR training, mDPR-seen. We also synthetically create mGEN training data as explained in § 2.2.1 by simply replacing answer entities with the corresponding ones in the target languages. The languages that are unseen by mDPR but are seen by mGEN are called mGEN-seen, and all other languages (i.e., included neither in mDPR nor mGEN training; 9 of the MKQA languages) unseen languages.

heading: 3.2 Baselines and Experimental Setting

text:
We compare CORA with the following strong baselines adopted from Asai et al. (2021).
Translate-test (MT + DPR). As used in most previous work (e.g., Asai et al., 2021), this method translates a question to English, extracts an answer in English using DPR, and then translates the answer back to the target language. The translation models are obtained from MarianMT (JunczysDowmunt et al., 2018) and trained on the OPUS-MT dataset (Tiedemann, 2012).
Monolingual baseline (BM25). This baseline retrieves passages solely from the target language and extracts the answer from the retrieved passages. Training neural network models such as DPR is infeasible with a few thousands of training examples. Due to the lack of training data in most of
3A full list of the language families and script types are in the appendix. 4Downloaded from https://archive.org/details/wikimediadownloads?and%5B%5D=
year%3A%222019%22. 5https://github.com/attardi/wikiextractor 6https://en.wikipedia.org/wiki/Category:Disambiguation_pages.
the target languages, we use a BM25-based lexical retriever implementation by Pyserini (Lin et al., 2021). We then feed the retrieved documents to a multilingual QA model to extract final answers.
MT+Mono. This baseline combines results from the translate-test method and the monolingual method to retrieve passages in both English and the target language. Following Asai et al. (2021), we prioritize predictions from the monolingual pipeline if they are over a certain threshold tuned on XOR-TYDI QA development set; otherwise we output predictions from the translate-test method.7
Closed-book baseline. This model uses an mT5-base8 sequence-to-sequence model that takes a question as input and generates an answer in the target language without any retrieval at inference time (Roberts et al., 2020). This baseline assesses the models’ ability to memorize and retrieve knowledge from its parameters without retrieving reference documents.
CORA details. For all experiments, we use a single retriever (mDPR) and a single generator (mGEN) that use the same passage embeddings. mDPR uses multilingual BERT base uncased,9 and the generator fine-tunes mT5-base. We found that using other pre-trained language models such as mBART (Liu et al., 2020) for mGEN or XLM-R (Conneau et al., 2020) for mDPR did not improve performance and sometimes even hurt performance. We first fine-tune mDPR using gold passages from Natural Questions, and then further fine-tune it using XOR-TYDI QA and TYDI QA’s gold passage data. We exclude the training questions in Natural Questions and TYDI QA that were used to create the MKQA or XOR-TYDI QA evaluation set. We run two iterations of CORA training (§ 2.2) after the initial fine-tuning. All hyperparameters are in Appendix § B.5.

heading: 4 Results and Analysis

heading: 4.1 Multilingual Open QA Results

text:
XOR-TYDI QA. Table 1 reports the scores of CORA and the baselines in XOR-TYDI QA. CORA, which only uses a single retriever and a single generator, outperforms the baselines and the previous state-of-the-art model on XOR-TYDI QA by a large margin across all 7 languages. CORA achieves gains of 24.8 macro-averaged F1 points over the previous state-of-the-art method (GMT+GS), which uses external black-box APIs, and 23.4 points over the concurrent anonymous work (SER).
MKQA. Tables 2 and 3 report the F1 scores of CORA and the baselines on over 6.7k MKQA questions with short answer annotations10 under seen and unseen settings. CORA significantly
7For the languages not supported by Pyserini, we always output translate-test’s predictions. 8We did not use larger-sized variants due to our computational budget. 9The alternative of XLM-RoBERTa (Conneau et al., 2020) did not improve our results.
10Following previous work in open QA but different from the official script of MKQA (Longpre et al., 2020), we disregard the questions labeled as “no answer”. As shown in our human analysis, it is difficult to prove an answer does not exist in the millions of multilingual documents even if the annotation says so.
outperforms the baselines in all languages by a large margin except for Arabic and English. Note that Longpre et al. (2020) report results in a simplified setting with gold reference articles from the original Natural Questions dataset given in advance, and thus their results are not comparable. CORA yields larger improvements over the translate-test baseline in the languages that are distant from English and with limited training data such as Malay (Ms; 27.8 vs. 12.6) and Hebrew (He; 15.8 vs. 8.9). The performance drop of the translate-test model from English (43.3 F1) to other languages indicates the error propagation from the translation process. BM25 performs very poorly in some low-resource languages such as Thai because of the lack of answer content in the target languages’ Wikipedia. MT+Mono underpeforms the MT+DPR baseline in MKQA since it is challenging to rerank answers from two separate methods with uncaliberated confidence scores. In contrast, CORA retrieves passages across languages, achieving around 30 F1 on a majority of the 26 languages.

heading: 4.2 Analysis

text:
Ablations: Impact of CORA components. We compare CORA with the following four variants to study the impact of different components. (i) mDPR1 + mGEN1 only trains CORA using the initial labeled, annotated data and measures the impact of the iterative training. (ii) DPR (trained NQ) + mGEN replaces mDPR with a multilingual BERT-based DPR trained on English data from Natural Questions (NQ), and encodes all passages in Cmulti. This configuration assesses the impact of cross-lingual training data. (iii) CORA, Cmulti={En} only retrieves from English during inference. This variant evaluates if English reference documents suffice to answer multilingual questions. (iv) mDPR+Ext.reader+MT replaces mGEN with an extractive reader model (Karpukhin et al., 2020) followed by answer translation. This variant quantifies the effectiveness of using a multilingual generation model over the approach that combines an extractive reader model with language-specific translation models. Note that for MKQA experiments, we sample the same 350 questions (∼5%) from the evaluation set for each language to reduce the computational cost over varying configurations.
Results in Table 4 show performance drops in all variants. This supports the following claims: (i) the iterative learning and data mining process is useful, (ii) mDPR trained with cross-lingual data substantially outperforms DPR with multilingual BERT trained on monolingual data only, (iii) reference languages other than English are important in answering multilingual questions, and (iv) a multilingual generation model substantially boosts the model performance.
Retrieval performance and relationship to the final QA performance. We evaluate CORA’s retrieval performance on MKQA using two recall metrics that measure the percentage of questions with at least one passage among the top 10 that includes a string in an answer set in the target language (RL@10) or in the union of answer sets from all languages that are available in MKQA (Rmulti@10). MKQA provides answer translations across 26 languages.
Table 5 reports retrieval results for mDPR and multilingual BERT-based DPR trained on NQ: DPR (NQ). This is equivalent to (ii) from the ablations. We observe that mDPR performs well in IndoEuropean languages with Latin script, even when the language is unseen. Interestingly, there is a significant performance gap between RL@10 and Rmulti@10 in languages with non-Latin script (e.g., Japanese, Russian, Chinese); this suggests that our model often uses relevant passages from other languages with Latin script such as English or Spanish to answer questions in those languages with non-Latin script. Our mDPR outperforms DPR (NQ) by a large margin in unseen languages with limited resources, which are consistent with the findings in Table 3. Nevertheless, we still see low performance on Khmer and Thai even with the Rmulti@10 metric. We also observe that passage and query embeddings for those languages are far from other languages, which can be further studied in future work. We provide a two-dimensional visualization of the encoded passage representations in the appendix.
Breakdown of reference languages. Fig. 3 breaks down retrieved reference languages for each target language. Our multilingual retrieval model often retrieves documents from the target language (if its reference documents are available), English, or its typologically similar languages. For example, mDPR often retrieves Spanish passages for Portuguese questions and Japanese passages for Chinese questions; while they are considered phylogenetically distant, Japanese and Chinese overlap in script.
To further evaluate this, we conduct a controlled experiment: we remove Spanish, Swedish and Indonesian document embeddings and evaluate CORA on related languages: Danish, Portuguese and
Malay. We observe performance drops of 1.0 in Danish, 0.6 in Portuguese, and 3.4 F1 points in Malay. This illustrates that while CORA allows for retrieval from any language in principle (many-to-many), cross-lingual retrieval from closer languages with more language resources is particularly helpful.
Error analysis and qualitative examples. Table 6 analyzes errors from CORA by manually inspecting 50 Japanese and Spanish wrong predictions from MKQA. We observe six major error categories: (a) retrieval errors, (b) generating correct answers in a different language (different lang), (c) incorrect answer generation (incorrect answer), (d) answer annotation errors (e.g., a correct alias isn’t covered by gold answers, or Wikipedia information is inconsistent with English.), and (e) ambiguous or underspecified questions such as “who won X this year” (underspecified q). The table shows that both in Japanese and Spanish, the retrieval errors are dominant. In Japanese, CORA often generates correct answers in English, not in Japanese (different lang).
Fig. 4 shows some qualitative examples. The first example shows an error in (b): mGEN is generating an answer in Russian, not in French though the answer itself is correct. This type of error happens especially when retrieved passages are in languages other than the target and English.
Human evaluation on cross-lingual retrieval results. To observe how cross-lingual retrieval between distant languages is actually helping, we sample 25 Norwegian questions for which Spanish passages are included among the top 10 retrieved results. As seen in Fig. 3, CORA retrieves Spanish (es) passages for 6.8% of the Norwegian (no) questions. A Spanish speaker judges if the retrieved Spanish passages actually answer the given Norwegian questions.11 We found that in 96% of the cases, the retrieved Spanish passages are relevant in answering the question. One such example is presented in Fig. 4 (the second example).
Human analysis on unanswerable questions. CORA retrieves passages from a larger multilingual document collection than the original human annotations. Thus, CORA may further improve the answer coverage over the original human annotations. MKQA includes questions that are marked as unanswerable by native English speakers given English knowledge sources. We sample 400 unanswerable Japanese questions whose top one retrieved passage is from a non-English Wikipedia article. Among these, 329 unanswerable questions are underspecified (also discussed in Asai and Choi, 2021). For 17 out of the 71 remaining questions, the answers predicted by CORA are correct. This finding indicates the significance of cross-lingual retrieval and potential room for improvement in annotating multilingual open QA datasets. The third example in Fig. 4 shows one of these cases.

heading: 5 Related Work and Broader Impacts

text:
English and non-English open QA. Despite the rapid progress in open QA (Chen et al., 2017; Karpukhin et al., 2020), most prior work has been exclusively on English (Lewis et al., 2020; Izacard and Grave, 2021b). Several prior attempts to build multilingual open QA systems often rely on machine translation or language-specific retrieval models (Ture and Boschee, 2016; Asai et al., 2021).
11During evaluation, we provide the original English questions from MKQA.
Lewis et al. (2020) and Guu et al. (2020) introduce retrieve-then-generate approaches. Izacard and Grave (2021a) introduce an iterative training framework that uses attention weights from a generator model as a proxy for text relevance scores. Tran et al. (2020) introduce CRISS, a self-supervised pre-training approach consisting of a parallel sentence mining module and a sequence-to-sequence model, which are trained iteratively. Several recent work such as Xiong et al. (2021) improves DPR by mining and learning with hard examples. Our work is the first work that introduces a unified multilingual system for many-to-many open QA, which is a challenging task requiring massive-scale cross-lingual retrieval and has not been addressed in prior work. We introduce an iterative training and data mining approach guided by filtering from an answer generation model to automatically extend annotated data available only in high-resource languages to low-resource. This approach contributes to significant performance improvements in languages without annotated training data.
Many-languages-one models. Several recent work introduces single multilingual models for many languages using pre-trained multilingual models such as mBERT or mT5 in many NLP tasks (e.g., entity linking: Botha et al., 2020; De Cao et al., 2021; semantic role labeling: Mulcaire et al., 2019b; Lyu et al., 2019; Fei et al., 2020; syntactic parsing: Mulcaire et al., 2019a; Kondratyuk and Straka, 2019). This work conducts the first large-scale study of a unified multilingual open QA model across many languages and achieves state-of-the-art performance in 26 typologically diverse languages.
Synthetic data creation for machine reading comprehension. Alberti et al. (2019) introduce a method of generating synthetic machine reading comprehension data by automatically generating questions and filtering them out by a trained machine reading comprehension model. Several studies augment multilingual machine reading comprehension training data by generating new questionanswer pairs from randomly sampled non-English Wikipedia paragraphs (Riabi et al., 2021; Shakeri et al., 2020). This work focuses on multilingual open QA, which involves not only machine reading comprehension but also cross-lingual retrieval. A similar augmentation method for machine reading comprehension can be applied to further improve the answer generation component in CORA.
Societal impacts. Our code and data are publicly available. CORA can perform open QA in unseen languages and can benefit society in building QA systems for low-resource languages, hence enabling research in that direction. Unlike previous models, CORA removes the necessity of external black-box APIs, and thus we can examine and address wrong answers due to model errors or misinformation present on Wikipedia. This would help us mitigate the potential negative impact from CORA or its subsequent models outputting a wrong answer when it is used by people who seek information.

heading: 6 Conclusion

text: To address the information needs of many non-English speakers, a QA system has to conduct crosslingual passage retrieval and answer generation. This work presents CORA, a unified multilingual many-to-many open QA model that retrieves multilingual passages in many different languages and generates answers in target languages. CORA does not require language-specific translation or retrieval components and can even answer questions in unseen, new languages. We conduct extensive experiments on two multilingual open QA datasets across 28 languages, 26 of which CORA advances the state of the art on, outperforming competitive models by up to 23 F1 points. Our extensive analysis and manual evaluation reveal that CORA effectively retrieves semantically relevant passages beyond language boundaries, and can even find answers to the questions that were previously considered unanswerable due to lack of sufficient evidence in annotation languages (e.g., English). Nonetheless, our experimental results show that the retrieval component still struggles to find relevant passages for queries in some unseen languages. Our analysis also showed that CORA sometimes fails to generate an answer in the target language. In future work, we aim to address these issues to further improve the performance and scale our framework to even more languages.

heading: Acknowledgement

text: This research was supported by NSF IIS-2044660, ONR N00014-18-1-2826, gifts from Google, the Allen Distinguished Investigator Award, the Sloan Fellowship, and the Nakajima Foundation Fellowship. We thank anonymous reviewers, area chairs, Eunsol Choi, Sewon Min, David Wadden, and the members of the UW NLP group for their insightful feedback on this paper, and Gabriel Ilharco for his help on human analysis.

title: One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval


summaries: This paper conducts a large-scale empirical study of ERM models tested in the domainBed pipeline, together with the empirical results tested, the authors also investigated the numerical values of several different measures, which allows the authors to further study the relationship between multiple different measures and the empirical performances. The empirical study suggests that there may be interesting connections between the target-entropy and the empirical performances. The paper candidly acknowledges its limitations of being purely empirical.

summaries: The paper provides a large scale study of out-of-distribution generalization measures. It first provides a brief introduction of the core theoretical foundations of the field. Then the paper trains 12000 models on DomainBed Benchmarks to examine the predictive power of theory-based measures. The paper argues that theory based measure fail to accurately capture the OOD generalization behavior. Following the methodology of Jiang 2019 et al, the paper examine how various empirical measures (sharpness, entropy, Fisher, ...) predict OOD generalization behavior. In the end, the paper lists the predictive measures and examines their performance details.

summaries: This work tried to find that which measures can be used for predicting the the out-of-domain generalization for the deep neural networks trained using Empirical Risk Minimization (ERM). The authors started from the domain adaptation theory of Ben-David, 2007 but found that it had limited ability. Then they explored many other measures, including Fisher based measures, Jacobian Norm based measures, Mixup based Measures, and entropy on target data. They found the best single factor led to around 0.70 Spearman's ρ and the joint use can boost the value to 0.80.

summaries: Prior work has found that empirical risk minimization performs unexpectedly well on domain generalization tasks. In this work, the authors try to explain this phenomenon. First, they use theory developed for domain adaptation to bound the target domain test error, finding that theory-based measures do not adequately predict the target domain test error, especially when target domain labels are not available. Next, the authors try a variety of empirical measures. They find that some measures outperform the theory, especially when combined with the in-domain test error.

abstractText: Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goal of this paper is to further understand this phenomenon. In particular, we study the extent to which the seminal domain adaptation theory of Ben-David et al. (2007) explains the performance of ERMs. Perhaps surprisingly, we find that this theory does not provide a tight explanation of the out-of-domain generalization observed across a large number of ERM models trained on three popular domain generalization datasets. This motivates us to investigate other possible measures—that, however, lack theory—which could explain generalization in this setting. Our investigation reveals that measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy are good predictors of the out-of-distribution generalization of ERM models. We hope that our work helps galvanize the community towards building a better understanding of when deep networks trained with ERM generalize out-of-distribution.

title: Natural gradient works efficiently in learning

venue: Neural Comput.,

authors:
M. Arjovsky
L. Bottou
I. Gulrajani
D. Lopez-Paz

title: Invariant risk minimization

authors:
S. Arora
R. Ge
B. Neyshabur
Y. Zhang

title: Stronger generalization bounds for deep nets via a compression approach

authors:
P. Bartlett
D.J. Foster
M. Telgarsky

title: Spectrally-normalized margin bounds for neural networks

authors:
S. Ben-David
J. Blitzer
K. Crammer
F. Pereira

title: Analysis of representations for domain adaptation

venue: Advances in Neural Information Processing Systems,

authors:
S. Ben-David
J. Blitzer
K. Crammer
A. Kulesza
F. Pereira
J. Vaughan

title: A theory of learning from different domains

venue: Machine Learning,

authors:
L. Carratino
M. Cissé
R. Jenatton
Vert
J.-P

title: On mixup regularization

authors:
C. Cortes
M. Mohri
A.M. Medina

title: Adaptation based on generalized discrepancy

venue: J. Mach. Learn. Res.,

authors:
A. Daniely
E. Granot

title: Generalization bounds for neural networks via approximate description length

authors:
H. Drucker
Y. Le Cun

title: Improving generalization performance using double backpropagation

venue: IEEE Trans. Neural Netw.,

authors:
G.K. Dziugaite
D.M. Roy

title: Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data

authors:
G.K. Dziugaite
A. Drouin
B. Neal
N. Rajkumar
E. Caballero
L. Wang
I. Mitliagkas
D.M. Roy

title: In search of robust measures of generalization

authors:
C. Fang
Y. Xu
D.N. Rockmore

title: Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias

venue: IEEE International Conference on Computer Vision,

authors:
Y. Ganin
E. Ustinova
H. Ajakan
P. Germain
H. Larochelle
F. Laviolette
M. Marchand
V. Lempitsky

title: Domain-Adversarial training of neural networks

authors:
M. Ghifary
W. Bastiaan Kleijn
M. Zhang
D. Balduzzi

title: Domain generalization for object recognition with multi-task autoencoders

authors:
A. Gretton
A. Smola
J. Huang
M. Schmittfull
K. Borgwardt
B. Schölkopf

title: Covariate shift by kernel mean matching

venue: Dataset shift in machine learning,

authors:
I. Gulrajani
D. Lopez-Paz

title: In search of lost domain generalization

authors:
D. Hendrycks
M. Mazeika
T. Dietterich

title: Deep anomaly detection with outlier exposure

authors:
D. Hendrycks
S. Basart
N. Mu
S. Kadavath
F. Wang
E. Dorundo
R. Desai
T. Zhu
S. Parajuli
M. Guo
D. Song
J. Steinhardt
J. Gilmer

title: The many faces of robustness: A critical analysis of Out-of-Distribution generalization

authors:
J. Hoffman
D.A. Roberts
S. Yaida

title: Robust learning with jacobian regularization

authors:
S. Jastrzebski
D. Arpit
O. Astrand
G. Kerg
H. Wang
C. Xiong
R. Socher
K. Cho
K. Geras

title: Catastrophic fisher explosion: Early phase fisher matrix impacts generalization

authors:
Y. Jiang
B. Neyshabur
H. Mobahi
D. Krishnan
S. Bengio

title: Fantastic generalization measures and where to find them

authors:
D. Krueger
E. Caballero
Jacobsen
J.-H
A. Zhang
J. Binas
R. Le Priol
A. Courville

title: Out-of-Distribution generalization via risk extrapolation (REx)

authors:
D. Li
Y. Yang
Song
Y.-Z
T.M. Hospedales

title: Learning to generalize: Meta-Learning for domain generalization

title: Deeper, broader and artier domain generalization

authors:
M. Mohri
A.M. Medina

title: New analysis and algorithm for learning with drifting distributions

authors:
B. Neyshabur
R. Tomioka
N. Srebro

title: Norm-Based capacity control in neural networks

authors:
B. Neyshabur
S. Bhojanapalli
D. McAllester
N. Srebro

title: Exploring generalization in deep learning

authors:
R. Novak
Y. Bahri
D.A. Abolafia
J. Pennington
J. Sohl-Dickstein

title: Sensitivity and generalization in neural networks: an empirical study

authors:
S.J. Pan
I.W. Tsang
J.T. Kwok
Q. Yang

title: Domain adaptation via transfer component analysis

authors:
A. Paszke
S. Gross
F. Massa
A. Lerer
J. Bradbury
G. Chanan
T. Killeen
Z. Lin
N. Gimelshein
L. Antiga
A. Desmaison
A. Köpf
E. Yang
Z. DeVito
M. Raison
A. Tejani
S. Chilamkurthy
B. Steiner
L. Fang
J. Bai
S. Chintala

title: PyTorch: An imperative style, High-Performance deep learning

authors:
B. Sun
K. Saenko

title: Deep CORAL: Correlation alignment for deep domain adaptation

title: An overview of statistical learning theory

authors:
D. Varga
A. Csiszárik
Z. Zombori

title: Gradient regularization improves accuracy of discriminative models

authors:
Y. Wu
E. Winston
D. Kaushik
Z. Lipton

title: Domain adaptation with Asymmetrically-Relaxed distribution alignment

venue: Proceedings of the 36th International Conference on Machine Learning,

authors:
S. Yan
H. Song
N. Li
L. Zou
L. Ren

title: Improve unsupervised domain adaptation with mixup training

authors:
C. Zhang
S. Bengio
M. Hardt
B. Recht
O. Vinyals

title: Understanding deep learning requires rethinking generalization

authors:
H. Zhang
M. Cisse
Y.N. Dauphin
D. Lopez-Paz

title: mixup: Beyond empirical risk minimization

venue: arXiv preprint arXiv:1710.09412,

authors:
H. Zhao
R.T. des Combes
K. Zhang
G.J. Gordon

title: On learning invariant representation for domain adaptation

heading: 1 Introduction

text:
Conventional wisdom in domain generalization was recently upturned by the work of Gulrajani & Lopez-Paz (2020) on the DomainBed benchmark. In their work, the authors use careful model selection and evaluation to show that models trained using empirical risk minimization (Vapnik, 1999) are able to achieve near state-of-the-art performance on a variety of popular benchmarks, surpassing the performance of most algorithms specialized for domain generalization (Arjovsky et al., 2019; Yan et al., 2020; Li et al., 2017a).
In a parallel research effort, there has been an increased interest in understanding when deep neural networks trained using empirical risk minimization generalize well in-domain (Jiang et al., 2019; Neyshabur et al., 2017; Dziugaite & Roy, 2017; Bartlett et al., 2017; Arora et al., 2018; Daniely & Granot, 2019). However, understanding when empirical risk minimizers are able to generalize out-of-distribution remains poorly understood. Answering this question would help us characterize the failure modes of domain generalization algorithms, and to develop specialized baselines that would allow us to break through the dominance of empirical risk minimization.
We argue that the out-of-distribution generalization performance of a classifier depends mainly on two factors. On the one hand, we expect good out-of-distribution generalization only for classifiers able to achieve good in-distribution generalization (also called source-domain validation accuracy). On the other hand, we expect out-of-distribution generalization to degrade as the discrepancy between the distributions of training and testing data increases, relative to the function family of interest. This reveals an interplay between the properties of the neural network, the used training data, and the
∗ Both D.L.P and D.J.S. contributed to the work equally 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
differences between training and testing regimes (Figure 1). Given all of these distinct ingredients, how does one build a coherent theory of domain generalization with strong predictive power for modern, over-parametrized neural networks?
The seminal work of Ben-David et al. (2007, 2010) lays down a theoretical foundation for the related setup of unsupervised domain adaptation. In particular, the theory in Ben-David et al. (2007, 2010) upper-bounds the out-of-distribution performance of an ERM based on a fixed feature space as the sum of two terms: (i) the source-domain validation accuracy, and (ii) the distance between the input distribution of source-domain and target-domain examples under the fixed feature space. This theory has found success in practice, improving both linear models (Ben-David et al., 2007) and deep neural networks (Ganin et al., 2015) in the setup of unsupervised domain adaptation. Algorithmically speaking, one can implement this theory by finding a feature representation under which (i) one can perform well on source-domain inputs, and (ii) it is not possible to distinguish source-domain inputs from target-domain inputs.
While the theory above deals with the setup of unsupervised domain adaptation (where algorithms have access to unlabeled target-domain data during training), here we are interested in the related but distinct domain generalization scenario where algorithms only have access to source-domain data during training. Therefore, to remain in the domain generalization setup, we will put the theory of Ben-David et al. (2007) to use from a different angle. Instead of training representations in such a way that encourages source-domain and target-domain inputs to be indistinguishable, our methodology consists in training a very large number of ERM models only using source-domain data, to later investigate if the models exhibiting good out-of-distribution generalization indeed learned a feature space under which it is difficult to tell apart source-domain and target-domain inputs.
Training over 12,000 deep neural network image classifiers using empirical risk minimization, using the standard DomainBed pipeline, we reveal that the theory of Ben-David et al. (2007) has limited power to predict when a classifier will generalize out-of-domain. We investigate this further by unpacking whether assumptions in the theory are violated or if the bounds are slack. Our findings suggest that in practice, while the assumptions from the theory are largely not violated in the learned feature spaces of ERMs, one can obtain good generalization despite being able to tell apart the source and target data, limiting the practical applicability of the theory to explain domain generalization. We investigate and provide more intuitions for this phenomenon in section 4.
In search of other correlates of domain generalization that might point the way to new theoretical approaches, we set out to build a catalog of measures that could predict the out-of-distribution generalization performance of our suite of ERM trained neural networks. In building such a catalog of generalization measures, we take inspiration from recent literature undertaking a similar effort to explain variance of in-domain generalization of deep neural networks (Jiang et al., 2019; Neyshabur et al., 2017; Dziugaite et al., 2020). After an exhaustive empirical evaluation, we find that several of the proposed measures are relatively strong predictors of out-of-distribution performance. These include measures related to predictive entropy, the Mixup criterion (Zhang et al., 2017), the norm of the Jacobian (Novak et al., 2018), and the Fisher information matrix (Amari, 1998).
To summarize, our work makes the following contributions:
• We perform a large scale empirical study testing the theory from Ben-David et al. (2007, 2010) on deep neural networks trained on the DomainBed (Gulrajani & Lopez-Paz, 2020) domain generalization benchmark.
• We find that the theory has limited ability to predict which ERMs will generalize out-ofdistribution.
• We perform an empirical evaluation of several other candidate measures for explaining domain generalization of neural networks, finding measures that outperform theoretically motivated quantities.

heading: 2 Background on Relevant Theory

text:
There is a rich line of work on the theory of domain adaptation, which focuses on learning and adapting when presented with test data drawn from a different input distribution from that seen during training (Ben-David et al., 2007; Mohri & Medina, 2012; Cortes et al., 2019). There are also a number of algorithms for domain adaptation which make use of unlabeled data to learn more invariant predictors, both for classical kernel machines (Pan et al., 2011; Gretton et al., 2009) as well as deep neural networks (Ganin et al., 2015). The results typically provide bounds on the test error in a target domain in terms of the test error of the model on a training domain, and a measure of the difference between the train and target input distributions in a feature space. Here we are interested in whether the theory can accurately predict out-of-distribution performance of an ERM.
In particular, we first focus on a classic result from Ben-David et al. (2007), which, given a fixed feature representation (R), derives generalization bounds for the test performance in terms of how well the model can distinguish source from target samples. This work focuses on the single-source (SS) setting, where a single source-domain is used for training. A central result from this work is a bound on the test error on the target domain ( T (h)), consisting of three distinct terms: the sourcedomain validation error ( S(h)), the so-called H-divergence between the source and target domains, and the joint source-target validation error when optimizing a single model for both environments, i.e. λ = minh∈H{ T (h) + S(h)}. In fact, in the theory, λ-closeness is assumed, and we will test this assumption later. We consider the predictive value of each of these three terms separately as well as their sum in our analysis. The H-divergence can be understood as related to and approximated by the best performance of a classifier in the function family H trained to distinguish samples from the source and target domains.
Subsequent work generalized this theory to multi-source (MS) training in Ben-David et al. (2010). Assuming an equal weighting of all training domains, the gap between the test errors in the target and source domains can be bounded above by the sum of two terms: the average λj , where λj = minh∈H{ T (h) + j(h)}, and the H∆H-divergence, similar to the H-divergence but for the symmetric difference hypothesis class. A full description of these terms can be found in the supplementary material.

heading: 3 Experiments

text: In this section we describe our methodology and experimental setup for addressing the problem of implicit domain generalization of ERMs. We first start with a brief overview of the DomainBed benchmark and then describe the collection of models we train in an effort to measure generalization.

heading: 3.1 DomainBed Benchmark and Training Setup

text:
DomainBed (Gulrajani & Lopez-Paz, 2020)1 is a recently released toolbox for domain generalization which provides implementations of various domain generalization algorithms such as Invariant Risk Minimization (Arjovsky et al., 2019), Variance Risk Extrapolation (Krueger et al., 2020), and metalearning based domain generalization (Li et al., 2017a) in addition to unsupervised domain adaptation approaches (Sun & Saenko, 2016; Ganin et al., 2015; Yan et al., 2020). It also provides various datasets such as RotatedMNIST (Ghifary et al., 2015), VLCS (Fang et al., 2013), and PACS (Li et al., 2017b). These datasets are provided in the form of multiple environments e = {(Xie, yie)}Ni=1, each
1https://github.com/facebookresearch/DomainBed (MIT License)
with a different training distribution pe(X) and (generally) the same conditional label distribution p(y∣X). Domain generalization algorithms are then trained on a subset of source environments Esource = {ejsource}Mj=1 and tested on target environments Etarget = {e j target}Kj=1.
An intriguing result from the DomainBed benchmark is that standard ERM training can match or outperform more specialized domain generalization algorithms. We would like to understand this phenomenon better by characterizing properties that distinguish ERMs (Empirical Risk Minimizers) which perform well from those which do not. We follow Gulrajani & Lopez-Paz (2020) in combining all the training environments into one source dataset S = ⋃e∈Etrain e used to train our ERMs. Testing is performed independently on each target environment.
We use the same network architectures as those used in the DomainBed paper (see supplemental information for more details). Overall our networks span shallow convolutional neural networks with 386K parameters (on the MNIST datasets) to residual networks with 25 million parameters (on VLCS and PACS). We prioritize a careful, exhaustive hyperparameter sweep with the above architectures, to obtain models with different levels of performance (Gulrajani & Lopez-Paz, 2020), over studying different architectures which is arguably a second order effect on this problem. Following Gulrajani & Lopez-Paz (2020), for each combination of dataset and training environments, we pick 100 random hyperparameter settings of batch size, learning rate, weight decay, and dropout (for resnet models). For both source S and target T , we hold out 50% of the data for validation. Since no training happens on T this means that we can use 50% of the data in T for computing generalization measures and the other 50% for assessing the out-of-distribution error of an an empirical risk minimizer ĥ, which we term as T (ĥ). Analogously, we denote the validation error of the model on the source domain as S(ĥ).

heading: 3.2 Trained Empirical Risk Minimizers (ERMs) and Implicit Domain Generalization

text: Using the above procedure, we train approximately 12,000 models on a compute cluster with Volta GPUs using PyTorch (Paszke et al., 2019). All models are trained for 5000 training steps and the model saved at the last step is used for analysis (Gulrajani & Lopez-Paz, 2020). fig. 2 shows the results of the sweep. We note that the source-domain validation accuracy or error S(ĉ) is a good candidate for explaining performance on domain generalization (see fig. 2). This is a common practice in the literature (Hendrycks et al., 2020), where model selection is often performed based on this criteria. However, we note that for a given source-domain validation accuracy, there is significant variation in the corresponding out-of-distribution accuracy, suggesting need for measures to characterize generalization more tightly.

heading: 4 Testing the Predictive Power of Theory-based Measures

text:
In this section, we empirically evaluate various elements of the theory discussed in section 2 on 12,000 neural networks trained via ERM on the DomainBed benchmark.
Our general methodology for testing the theory involves two major components. Firstly, we estimate the tightness in the bounds from Ben-David et al. (2007) empirically, and secondly, we study if the terms from the theory are predictive of performance. As explained in section 2, given D̃S , the set of features extracted on a source domain, and D̃T , the set of features extracted on the target domain, the
theory provides bounds on the target error T (c) in terms of the source error S(c), H-divergence (dH(D̃S , D̃T )) and closeness of the domains (λ) (both computed in the feature space of a model):
T (c) ≤ S(c) + dH(D̃S , D̃T ) + λ, (1)
Estimation Considerations: In order to test how well the theory works in practice, we need to estimate the H and the H∆H-divergences from finite samples. We construct and empirically study the variance of estimators for both in the supplementary material, finding that estimates of H∆Hdivergence have a lower SNR than H-divergence (for which there are known convergence guarantees when H has a finite VC-dimension). However, for completeness we report results with both. How loose are the bounds? In fig. 3 we plot for each divergence measure and dataset, the right hand side of the bound in eq. (1). This reveals several interesting observations. Firstly, when the bounds, which are of the form “divergence measure + λ + S(ĉ)” (see eq. (1) for an example) are loose (first box-plot in each figure) (where looseness is defined as the difference between right hand side and left hand side of eq. (1)) it is not because the representations R entailed by the networks do not yield domains which are λ close but because the divergence between the source and target domains is large. Thus, in principle ERMs seem to yield feature spaces from which one can support the learning of both target and source domain labels. However, ERMs do not appear to learn feature spaces which implicitly align the source domain to the (unseen) target domain in the domain generalization setting. In unsupervised domain adaptation (uDA) one could enforce the divergence to be small (Ganin et al., 2015) (since uDA assumes access to the target domain inputs D̃T ), which could result in tighter bounds on test error, as long as the domains remain lambda-close. However, recent work (Zhao et al., 2019; Wu et al., 2019) shows that enforcing a low divergence can yield representations which are no longer lambda close, rendering the theory less useful in practice from an uDA perspective. Thus, having a low divergence seems neither sufficient nor necessary for generalization.
Our goal is not just to understand how tight or loose the bounds are, but to check how well the measures computed based on them correlate with target-domain test error. What governs the regimes in which the bounds correlate well with test error T (ĥ)? Do tighter bounds lead to stronger correlation with test error? We find that this is not necessarily the case, for example, on RotatedMNIST with H∆H-divergence-MS based bound (H∆H-divergence-MS + λ + S(ĉ)) (where the gap is very low (fig. 3, right, yellow-boxes) the Spearman’s ρ with observed generalization T (ĥ) is 0.422 while on VLCS it is 0.567. Thus, it is not just how biased upper bound is, but also the variance which can play a major role in predicting generalization.
Why do variations in H-divergence not correlate well with observed generalization despite λcloseness? We illustrate with a simple example (fig. 4) how one can obtain good generalization despite being very far from the source domain in terms of H-divergence. In such a regime, one can obtain a high H-divergence even when the classifier learnt on the source domain generalizes well to the target domain. Of course, the theory accounts for this in principle, since the bound on the target error is not violated. However, it is not useful in practice for predicting generalization despite the λ-closeness assumption being satisfied.
Is the theory useful for predicting generalization? In order to use the bounds in a more practical setting, we must recall that we do not have access to the test labels from the target domain, and thus cannot compute λ-closeness. Instead, we consider a biased estimate of the bound, namely S(ĥ) +
H-divergence, for the H-divergence based measures as an illustration. Note that one can understand λ-closeness as an assumption in the original theory, as opposed to something that is intended to be empirically computed. Thus, in this section we compute practically accessible components of the theory and measure correlation with actual error. Overall, we find that the Spearman’s ρ across conditions drops substantially without accounting for λ-closeness. For example, for H∆Hdivergence-MS based bound the Spearman’s ρ with generalization drops from 0.496 to 0.136 on removing λ-closeness. Similar trends can be seen for H∆H-divergence-SS on removing λ from the bound, which drops from 0.509 to 0.152 and H-divergence which goes from 0.215 to 0.027. In a sense, this is not very surprising, since λ-closeness uses the target-domain labels which provides significant signal for predicting the target domain error. However, this indicates that the theory cannot be used to great effect for predicting generalization in practice. Finally notice that among these measures, one performs much better generally when using H∆H-divergence-MS or H∆H-divergence-SS than when using H-divergence.

heading: 5 Exploring Empirical Measures

text:
We have seen that the classic theory of domain generalization does not succeed at providing measures that are strongly predictive of out-of-distribution generalization ( T (ĉ)) in practice. Thus we next turn to exploring new measures that may provide better predictive power. We emphasize that our goal here is not to formulate highly predictive measure for use in practice, but rather to point towards promising directions for future theoretical investigations.
In pursuit of these goals, given the heterogeneity of the environments (see fig. 2), we make the following methodological choices: (1) we seek good predictors of generalization for each combination of dataset and environment (which we term a condition) and aggregate across conditions to find measures which are reliable across different conditions, (2) we learn both separate regressors for each condition and a single regressor across all conditions. In all cases, we perform Ridge regression with leave-one-out cross validation for all the analysis. Across all regressions, we use a train, test split of (0.7, 0.3). We use the coefficient of determination R2 as a metric which indicates the fraction of the variance of the target variable explained by the regression, and report the average R2 of a predictor across all conditions of dataset/environment pairs. Following the methodology of Jiang et al. (2019), we also compute and emphasize Spearman’s rank correlation coefficient (Spearman’s ρ) to quantify how well each measure predicts which ERMs will generalize out-of-distribution.

heading: 5.1 Description of selected measures

text:
Here we describe select measures that quantify the degree to which the model is sensitive to changes in the distribution of inputs. Here we briefly summarize a few of the most performant measures we find. A full list can be found in the supplemental information, including variants of many measures. First, we briefly introduce measures based on the Fisher information of the model and how it changes
when computed on samples from the source or target environment. Then we study the norm of the Jacobian of the model computed on source or target environment samples. Finally, we introduce a measure based on the entropy of the model’s predictions on source and target samples.
Fisher Based Measures. The Fisher information (Amari, 1998) measures the sensitivity of model outputs to changes in the parameters. We expect that models will generalize well out-of-distribution when the parameter directions that affect model outputs are similar when computed on train and test input samples. Let the model output for class i on an input sample xn be hθ,i(xn), where θ denotes the model parameters. The Fisher information is then defined as I = 1
N A T A, where
Aµ,(in) = 1√ hθ,i(xn) ∂hθ,i(xn) dθµ . Our generalization measures based on the Fisher information compare the Fisher obtained on the source training dataset IStr to the Fisher obtained on the target data ITtr . Note that the Fisher, unlike the Hessian does not require labeled data and thus can be computed on the unlabeled images. For details, see the supplementary material.
Jacobian Norm Based Measures. The input-output Jacobian, a measure of sensitivity of the model output to changes in the input, is defined as Ji,α =
∂hi(x) ∂xα
. Intuitively, a model that is less sensitive to changes in the input will generalize better. Thus, it has been studied in the context of robust learning (Hoffman et al., 2019) and in-distribution generalization (Novak et al., 2018), where it was found to be predictive of generalization at the level of individual test points. Similar to Novak et al. (2018), we compute the Frobenius norm of the Jacobian. See supplementary material for full details.
Mixup Based Measures. Mixup (Zhang et al., 2018) was proposed as a more robust alternative to ERM that encourages the learned function to be smooth, and has been shown to smooth the network’s input-output Jacobian (Carratino et al., 2020). Here we use mixup as a generalization measure, and use the model’s score function h instead of the labels y for interpolation. Our motivation is the same as that for the Jacobian: if the function is less smooth around target examples, the network should not generalize as well. Given a neural network h(X), a dataset T , and λ ∼ Beta(α, α), we compute the Mixup measure as: 1∣T ∣ ∑Xi∈T ,Xj∈T ;i≠j (λh(Xi) + (1 − λ)h(Xj) − h(λXi + (1 − λ)Xj))
2. See supplemental material for a detailed algorithmic description for computing this measure.
Entropy on Target Data. We also compute the output-entropy of the neural network h on the target domain data Ttr (Entropy-Target): 1∣Ttr∣ ∑Xi∈Ttr ∑ Y j=1 − log (h(Xi)[j]) ⋅ h(Xi)[j]. Entropy-Source is defined analogously but on source domain samples.

heading: 5.2 Results with Single Measures

text:
We provide a representative sampling of the results for the measures we study in this section, while deferring to the appendix for more detailed results spanning all the measures that we study. For this section, we first computed the Spearman’s ρ for each of the measures with target-domain error, T (ĉ), and selected the measure from each family of related measures (say, those related to the Jacobian for example) which has the highest correlation. As a pre-processing step, we canonicalize the measures to always have a positive correlation with T (ĉ) (see appendix for more details). Which measures outperform the theory based measures H∆H-divergence-SS and Hdivergence? In terms of Spearman’s ρ, we find that Entropy-Source (0.598), Entropy-Target (0.734), Jacobian (0.420), MMD-Gaussian (0.283), and Fisher Align (0.441) all outperform the theory inspired measures from section 4, where the best performing measure was H∆H-divergence-SS at Spearman’s ρ of 0.152 table 1. Intriguingly, Entropy-Target (0.734) outperforms in-domain test error, S(ĉ), (0.712) in terms of Spearman’s ρ, while using R2 it is slightly worse (0.608 v.s. 0.582). Practically analyzing the performance of Entropy-Target. We next analyze the performance of Entropy-Target across the different datasets. We find that Entropy-Target outperforms S(ĉ) on the RotatedMNIST dataset (0.853 v.s. 0.744 Spearman’s ρ), while on the VLCS and PACS datasets S(ĉ) is better, with 0.625 v.s. 0.557 Spearman’s ρ on VLCS and 0.732 v.s. 0.662 Spearman’s ρ on PACS. One plausible explanation for the strong performance in this regime is shown in fig. 5. Consider the case where we train with environments with rotation 30-75 degrees in RotatedMNIST (as in fig. 1). In such a situation S(ĉ) cannot distinguish between the expected performance on the 0 degree and 15 degree test environments.
Table 1: Different measures (rows) evaluated on either Spearman’s ρ or R2 metric between the measure and T (ĉ) (columns) where each measure is either used Alone as a predictor or in conjunction with in-domain test error, S(ĉ), and is allowed to use only one set of parameters across all datasets and source environments (Joint) or allowed to set parameters differently for different environments (Stratified). We report per-environment correlation averaged across environments. Approaches which perform better than source-domain validation accuracy are shown in bold, while approaches
which perform better than the theory based measures are highlighted (p-values for all the results below are orders of magnitude lower than performance numbers, and are thus dropped).
However we see in fig. 5 that the entropy distributions across different models computed on the 0 and 15 degree target environments are quite different, with higher confidence on the 15 degree test environment where better generalization occurs.
What contributes to the performance of Entropy-Source? We find that across all models, the Entropy-Source (computed on the source training set) is highly correlated with S(ĉ) (e.g. Spearman’s ρ of 0.857 on RotatedMNIST). This suggests that the performance of Entropy-Source is likely because of its relationship to S(ĉ). This also potentially explains why using both S(ĉ) and Entropy-Source jointly table 1 has the same performance as using S(ĉ) alone (0.712 Spearman’s ρ).
Performance of measures split across datasets. In fig. 6, we find that for many measures the correlation has the same sign across the different datasets, which is encouraging and indicates that the proposed measures might be useful to understand generalization. Overall we find that EntropySource, Entropy-Target, H-divergence (train), Fisher Align, Mixup, L2-Path Norm, MMD-Gaussian are reliable in terms of the sign of the correlation. However, H-divergence (train) and L2-Path Norm have a high variance across datasets, which is not ideal.
Do Jacobian or Fisher measures perform better? While both Jacobian and Fisher Align have similar Spearman’s ρ values overall (around ∼ 0.4) we find that the Jacobian, while highly performant
on RotatedMNIST with Spearman’s ρ of 0.83 across environments, performs quite poorly on PACS and VLCS (Spearman’s ρ ∼ 0), and thus may not be a reliable measure. Previous work has utilized the Jacobian (Novak et al., 2018) on four different image datasets (CIFAR10, FashionMNIST, MNIST, CIFAR100) and found it to be predictive of generalization at the level of individual test points coming from the same distribution. However, our results illustrate that in the out-of-distribution setting, the Jacobian is less reliable. In contrast, the measure Fisher Align is more stable across datasets (Spearman’s ρ of 0.368 on VLCS, 0.212 on PACS and 0.585 on RotatedMNIST). Our intuition for Fisher Align is that since the eigenvectors of the Fisher indicate the directions in parameter space in which the model is most sensitive, if these directions match on a target distribution, one is more likely to generalize.

heading: 5.3 Results with Compound Measures

text:
In the previous sub-section we studied each of the proposed measures in isolation. Next, we study if combining the measures with in-domain test error, S(ĉ), can yield better performance. This is similar to the theory, which bounds the gap between target and source domain validation errors. In each case, we fit a regression model of the form α S(ĉ) + β ∗ feature to predict T (ĉ) and then use the predictions to compute Spearman’s ρ. However, we also report R2 of the regression in table 1.
Joint model across all environments. We first perform the regression (as explained in section 3) across all datasets and environments jointly (table 1, “Joint” column). This does not allow the model to incorporate the specific environment in the weights of the regressor, but instead insists on the same predictor or “rule” across all environments. This allows a direct comparison to the experiments we discussed in the previous sub-section. We find that in this setup, only Entropy-Target, Jacobian and MMD-Mean-Cov when used in conjunction with S(ĉ) improve Spearman’s ρ (table 1), indicating that these measures might be complementary to S(ĉ). Stratified models for each environment. We next allow the regressor to choose a new set of parameters for each environment. In such a setting, it is impossible to perform worse than S(ĉ), since one can always recover a solution which places 0 weight on the second feature if that is indeed the best solution to be found. As expected, table 1 reveals that all measures improve in conjunction with S(ĉ) in this setting. However, it is still instructive to study which measures improve the performance of S(ĉ) the most, relative to others. We find that MMD-Gaussian, MMD-Mean-Cov, Jacobian, Entropy-Target all lead to Spearman’s ρ values greater than 0.80 in this setting, suggesting them to be promising candidates to pursue further in this context.
Which is better, L2-Path Norm or Mixup? While L2-Path Norm appears to have a higher Spearman’s ρ than Mixup in the Alone setting (table 1), it appears that most of the benefit from L2-Path Norm comes from it being a good predictor of in-distribution generalization (Neyshabur et al., 2015). When paired with S(ĉ), even in the stratified setting, we find that L2-Path Norm has the same performance as S(ĉ) (0.712 Spearman’s ρ), whereas Mixup improves S(ĉ) from 0.712 to 0.749 Spearman’s ρ. Similar trends can be observed for R2 as well.

heading: 6 Related work and Limitations

text:
Our work first sets out to test the seminal theory around unsupervised domain adaptation (BenDavid et al., 2007, 2010), which aims to understand when an approach is expected to generalize to test domains different from the training domain. We provide an empirical evaluation of some of the bounds and theory, and surprisingly find that less principled approaches based on the Fisher information and Jacobian Norm perform better than bounds inspired by domain adaptation theory.
Most related to our current work is the work on DomainBed (Gulrajani & Lopez-Paz, 2020) which observed that more careful experimental choices lead to ERMs that are competitive with state of the art domain generalization methods. Our work seeks to understand what characterizes ERMs that generalize, with our results suggesting the entropy of the output distribution, norm of the Jacobian, and measures based on the Fisher being promising candidates for future research.
Other related work (Zhao et al., 2019; Wu et al., 2019) finds that the theory of unsupervised domain adaptation (Ben-David et al., 2007, 2010) when used to minimize domain divergences (Ganin et al., 2015) is not always useful in practice, in terms of reducing target domain error. Our work is different in that we study domain generalization (as opposed to adaptation) where one cannot minimize the
domain divergences explicitly. Correspondingly, we uncover that the theory is not useful in practice for predicting generalization as opposed to enforcing it.
Our methodological approach also takes inspiration from Jiang et al. (2019); Neyshabur et al. (2017) which aims to understand how overparameterized deep networks are able to achieve good generalization, motivated in part by their ability to fit random labels (Zhang et al., 2016). We implement some of the best performing measures from the large-scale empirical study of Jiang et al. (2019) for the in-domain setting, and employ rank correlation as a measure of performance following this work. We believe it is a fruitful intellectual and practical challenge to understand the role of implicit regularization (Neyshabur et al., 2015) in the domain generalization setting as well.
Previous work (Hoffman et al., 2019; Drucker & Le Cun, 1992; Varga et al., 2017) has studied domain generalization using the Jacobian norm as a regularizer. While their aim is to build a more robust model for supervised learning, our goal is in characterizing if an ERM trained without any regularization can be shown to be implicitly learning smooth functions, and therefore help aid understanding of the implicit domain generalization phenomenon. Novak et al. (2018) show that the Jacobian norm is useful for predicting generalization at the level of individual test points for in-domain classification. Here we extend their experimental setting to the out-of-distribution case, and show that the Jacobian is useful at the population level. Jastrzebski et al. (2020) recently studied the role of the evolution of the trace of the Fisher information over the course of training and its impact on in-domain generalization and proposed a regularizer based on this insight. We utilize the Fisher for the domain generalization setting. In contrast to Jastrzebski et al. (2020) who study the trace of the Fisher, our best Fisher-based method uses the eigenvectors for training and test.
Finally, predicting out-of-distribution generalization is related to out-of-distribution detection, but there is an important difference. While in out-of-distribution detection the task is to flag out-ofdistribution examples that do not belong to any of the classes observed during training (Hendrycks et al., 2018), in out-of-distribution generalization we assume that test instances always belong to one of the classes of interest, albeit they are generated from a different distribution during test time.
Limitations: Our work is a step towards building an understanding of when networks trained with ERM generalize to novel target domains. However, our study is empirical, which limits the generality of our conclusions. We hope this effort spurs future work on theoretical aspects of this problem. While we follow the model architectures of Gulrajani & Lopez-Paz (2020) to obtain a large set of pretrained models, studying the impact of architectures in addition to hyperparameters is also an interesting line of study.

heading: 7 Conclusion

text:
In this work, we studied the domain generalization performance of ERM trained deep neural networks. Surprisingly, we found that adapting theory from unsupervised domain adaptation was of limited use in predicting how well models generalize out-of-distribution. We found that the assumptions of the theory were satisfied, in particular λ-closeness, yet the practically accessible components of the theory do not provide strong predictions about which models will generalize. In an effort to discover other correlates of domain generalization, we studied the performance of a suite of novel empirical measures, finding a number of promising candidates. Our work provides a critical first step for understanding the implicit domain generalization properties of ERMs and points towards promising future theoretical investigations into strongly predictive measures of domain generalization.
Social Impact
While most work that improves the performance of machine learning models can be leveraged by applications that have negative societal impact, we believe that the results of this paper actually take steps in a positive direction for the deployment of machine learning models in broader society. In particular, our work aims to understand when models that were not trained to be robust to domain shift break down or actually succeed on novel test domains. A better understanding and ability to predict model behavior in such situations should help in building more trustworthy machine learning methods that are less likely to result in unintended consequences when deployed on novel data. We emphasize that our work does not propose a new algorithm for domain generalization but rather attempts to predict when previously trained models can be trusted to perform in novel domains.

heading: Acknowledgments

text:
We would like to thank Kamalika Chaudhuri, Pascal Vincent, Ricardo Monti, Edward Grefenstette, Yann Dubois and numerous other colleagues at Facebook for discussions and feedback on the paper. We also thank Kamalika for detailed feedback on a draft of the paper.
Funding Transparency Statement
D.J.S was partially supported by the NIH under award R01EB026943, the NSF through the Center for the Physics of Biological Function (PHY-1734030), and as a Simons Investigator in the Mathematical Modeling of Living Systems (MMLS).

title: An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers