summaries: The paper attempts to provide a solution to an interesting problem of cross language retrieval based question answering in a zero shot (without requirement of any language-specific annotated training data). Authors propose a new DPR algorithm that to be able to retrieve documents relevant to the question across languages (for multiple languages). The paper extends retrieve-the-generate based open domain QA system to a cross-lingual setting.  This paper addresses the multilingual open QA task by combining a cross-lingual passage retrieval algorithm with a multilingual autoregressive generation model. The approach is thus capable of answering questions where the answer and the question are in different languages.  The authors extended existing annotated data using an iterative training approach that improves retrieval for languages with limited resources. The experimental results showed that the proposed method (CORA) outperformed the previous state of the art on multilingual open question answering benchmarks across 26 languages. This paper studies the task of multilingual Open-Domain Question Answering.  The task setting requires building a system that can answer natural language questions in any language on an open set of domains. Models are required to answer in the same language the question was asked in.  The authors tackle the task using a fairly conventional retrieve-and-read ODQA model, with the necessary modifications to make both the retriever and reader component language-agnostic, by exploiting multi-lingual pretrained models, training on available multilingual training data, and using and a clever data augmentation setup to increase multilingual training data.   The proposed solution “CORA”, operates over wikipedias in many different languages.   The retrieval component “MDPR” is a multi-lingual extension of DPR, using MBERT. This model can retrieve evidence documents across different languages, rather than having to match the language of the question. This means that a question expressed in a low-resource language can benefit from retrieving evidence from high resource languages. It also allows for a question written in one language on a cultural topic associated with a different language to retrieve from the most appropriate language’s wikipedia.   The reader “mGEN” is a generative reader, implemented using mT5, that takes as input the concatenation of the top-K retrieved passages from the retriever (which may be several different languages) and the question, and generates the answer in the question language, token by token.   The main difficulty of the proposed solution is how to train the MDPR and MGEN model components, given the limited multilingual raining data. The initial model is trained with NQ (english) and TyDi QA and Xor-TiDi QA (multilingual data). To increase the amount of multilingual training data and language coverage,  the authors introduce a data augmentation strategy where the initial model is used to harvest potential additional data. This exploits the wikipedia “language links” resource. This iterative approach works by retrieving passages, looking up passages on the same topic in other languages using “language links”, and using the reader to see whether these can be used to generate the correct answer - if they can, they are added as additional training data for the next round. Another expansion mechanism exploits the fact that most answers are wikipedia entities, and can be translated using annotations available in the wikipedia “language links” resource.  CORA is thoroughly evaluated on TyDI QA and an evaluation-only dataset called MKQA, and performs very strongly, improving the state-of-the-art by a wide margin in almost every setting.. Thorough ablations give insights and quantify the the effects of the system, and demonstrate zero-shot question answering abilities    This paper presents a multilingual dense passage retrieval model called mDPR to deal with the multilingual/cross-lingual IR and QA tasks. The mDPR part is firstly trained using existing labeled multilingual Q-P pairs. Then, possible passage pairs are selected from a given multilingual corpus based on the trained mDPR and used to train an answer generation model called mGEN. Next, given each question, the trained mDPR is further used to retrieve more possible passage candidates in other languages from Wikipedia corpus, where the retrieved passages that can lead to correct answers are considered as positive instances, the others as negative instances. With these expanded new question-passage pairs, the training of mDPR is performed again. The above procedure is conducted in an iterative manner. Evaluations are conducted on TyDi QA and MKQA in the open-domain QA setting. Several baselines are included and compared. The key contribution is to do DPR in the multilingual/cross-lingual setting.
abstractText: We present Cross-lingual Open-Retrieval Answer Generation (CORA), the first unified many-to-many question answering (QA) model that can answer questions across many languages, even for ones without language-specific annotated data or knowledge sources. We introduce a new dense passage retrieval algorithm that is trained to retrieve documents across languages for a question. Combined with a multilingual autoregressive generation model, CORA answers directly in the target language without any translation or in-language retrieval modules as used in prior work. We propose an iterative training method that automatically extends annotated data available only in high-resource languages to low-resource ones. Our results show that CORA substantially outperforms the previous state of the art on multilingual open QA benchmarks across 26 languages, 9 of which are unseen during training. Our analyses show the significance of cross-lingual retrieval and generation in many languages, particularly under low-resource settings. Our code and trained model are publicly available at https://github.com/AkariAsai/ CORA.
authors.name: Akari Asai
authors.name: Xinyan Yu
authors.name: Jungo Kasai
authors.name: Hannaneh Hajishirzi
references.authors: Chris Alberti Daniel Andor Emily Pitler Jacob Devlin Michael Collins.
references.title: Synthetic QA corpora generation with roundtrip consistency
references.venue: ACL.
references.authors: Akari Asai Eunsol Choi.
references.title: Challenges in information seeking QA: Unanswerable questions and paragraph retrieval
references.venue: ACL.
references.authors: Akari Asai Jungo Kasai Jonathan H. Clark Kenton Lee Eunsol Choi Hannaneh Hajishirzi.
references.title: XOR QA: Cross-lingual open-retrieval question answering
references.venue: NAACL.
references.authors: Jan A. Botha Zifei Shan Daniel Gillick.
references.title: Entity Linking in 100 Languages
references.venue: EMNLP.
references.authors: Danqi Chen Adam Fisch Jason Weston Antoine Bordes.
references.title: Reading Wikipedia to answer open-domain questions
references.venue: ACL.
references.authors: Jonathan H. Clark Eunsol Choi Michael Collins Dan Garrette Tom Kwiatkowski Vitaly Nikolaev Jennimaria Palomaki.
references.title: TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages
references.venue: TACL.
references.authors: Alexis Conneau Kartikay Khandelwal Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov.
references.title: Unsupervised cross-lingual representation learning at scale
references.venue: ACL.
references.authors: Nicola De Cao Ledell Wu Kashyap Popat Mikel Artetxe Naman Goyal Mikhail Plekhanov Luke Zettlemoyer Nicola Cancedda Sebastian Riedel Fabio Petroni
references.title: Multilingual autoregressive entity linking
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova.
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.venue: NAACL.
references.authors: Hao Fei Meishan Zhang Donghong Ji.
references.title: Cross-lingual semantic role labeling with highquality translated training corpus
references.venue: ACL.
references.authors: Daniel Gillick Sayali Kulkarni Larry Lansing Alessandro Presta Jason Baldridge Eugene Ie Diego Garcia-Olano.
references.title: Learning dense representations for entity retrieval
references.venue: CoNLL.
references.authors: Kelvin Guu Kenton Lee Zora Tung Panupong Pasupat Ming-Wei Chang.
references.title: Realm: Retrieval-augmented language model pre-training
references.venue: ICML.
references.authors: Gautier Izacard Edouard Grave.
references.title: Distilling knowledge from reader to retriever for question answering
references.venue: ICLR.
references.authors: Gautier Izacard Edouard Grave.
references.title: Leveraging passage retrieval with generative models for open domain question answering
references.venue: EACL.
references.authors: Marcin Junczys-Dowmunt Roman Grundkiewicz Tomasz Dwojak Hieu Hoang Kenneth Heafield Tom Neckermann Frank Seide Ulrich Germann Alham Fikri Aji Nikolay Bogoychev André F.T. Martins Alexandra Birch.
references.title: Marian: Fast neural machine translation in C++
references.venue: ACL (System Demonstrations).
references.authors: Vladimir Karpukhin Barlas Oğuz Sewon Min Ledell Wu Sergey Edunov Danqi Chen Wen-tau Yih.
references.title: Dense passage retrieval for open-domain question answering
references.venue: EMNLP.
references.authors: Diederik P Kingma Jimmy Ba.
references.title: Adam: A method for stochastic optimization
references.venue: ICLR.
references.authors: Dan Kondratyuk Milan Straka.
references.title: 75 languages, 1 model: Parsing Universal Dependencies universally
references.venue: EMNLP.
references.authors: Tom Kwiatkowski Jennimaria Palomaki Olivia Redfield Michael Collins Ankur Parikh Chris Alberti Danielle Epstein Illia Polosukhin Jacob Devlin Kenton Lee Kristina Toutanova Llion Jones Matthew Kelcey Ming-Wei Chang Andrew M. Dai Jakob Uszkoreit Quoc Le Slav Petrov.
references.title: Natural Questions: A benchmark for question answering research
references.venue: TACL.
references.authors: Kenton Lee Ming-Wei Chang Kristina Toutanova.
references.title: Latent retrieval for weakly supervised open domain question answering
references.venue: ACL.
references.authors: Patrick Lewis Ethan Perez Aleksandara Piktus Fabio Petroni Vladimir Karpukhin Naman Goyal Heinrich Küttler Mike Lewis Wen-tau Yih Tim Rocktäschel
references.title: Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS
references.authors: Jimmy Lin Xueguang Ma Sheng-Chieh Lin Jheng-Hong Yang Ronak Pradeep Rodrigo Nogueira
references.title: Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations
references.authors: Jimmy Lin Rodrigo Nogueira Andrew Yates
references.title: Pretrained transformers for text ranking: BERT and beyond
references.authors: Yankai Lin Haozhe Ji Zhiyuan Liu Maosong Sun.
references.title: Denoising distantly supervised open-domain question answering
references.venue: ACL.
references.authors: Yinhan Liu Jiatao Gu Naman Goyal Xian Li Sergey Edunov Marjan Ghazvininejad Mike Lewis Luke Zettlemoyer.
references.title: Multilingual denoising pre-training for neural machine translation
references.venue: TACL.
references.authors: Shayne Longpre Yi Lu Joachim Daiber
references.title: MKQA: A linguistically diverse benchmark for multilingual open domain question answering
references.authors: Chunchuan Lyu Shay B. Cohen Ivan Titov.
references.title: Semantic role labeling with iterative structure refinement
references.venue: EMNLP.
references.authors: Sewon Min Jordan Boyd-Graber Chris Alberti Danqi Chen Eunsol Choi Michael Collins Kelvin Guu Hannaneh Hajishirzi Kenton Lee Jennimaria Palomaki Colin Raffel Adam Roberts Tom et al Kwiatkowski.
references.title: NeurIPS 2020 EfficientQA competition: Systems, analyses and lessons learned
references.venue: PMLR.
references.authors: Sewon Min Danqi Chen Hannaneh Hajishirzi Luke Zettlemoyer.
references.title: A discrete hard em approach for weakly supervised question answering
references.venue: EMNLP.
references.authors: Phoebe Mulcaire Jungo Kasai Noah A. Smith.
references.title: Low-resource parsing with crosslingual contextualized representations
references.venue: CoNLL.
references.authors: Phoebe Mulcaire Jungo Kasai Noah A. Smith.
references.title: Polyglot contextual representations improve crosslingual transfer
references.venue: NAACL.
references.authors: Arij Riabi Thomas Scialom Rachel Keraron Benoît Sagot Djamé Seddah Jacopo Staiano.
references.title: Synthetic data augmentation for zero-shot cross-lingual question answering
references.venue: EMNLP.
references.authors: Adam Roberts Colin Raffel Noam Shazeer
references.title: How much knowledge can you pack into the parameters of a language model
references.authors: Siamak Shakeri Noah Constant Mihir Sanjay Kale Linting Xue
references.title: Towards zero-shot multilingual synthetic question and answer generation for cross-lingual reading comprehension
references.authors: Saku Sugawara Kentaro Inui Satoshi Sekine Akiko Aizawa
references.title: What makes reading comprehension questions easier
references.authors: Jörg Tiedemann.
references.title: Parallel data, tools and interfaces in OPUS
references.venue: LREC.
references.authors: Jörg Tiedemann Lars Nygaard.
references.title: The OPUS corpus - parallel and free
references.venue: LREC.
references.authors: Jörg Tiedemann Santhosh Thottingal.
references.title: OPUS-MT — Building open translation services for the World
references.venue: EAMT.
references.authors: Chau Tran Yuqing Tang Xian Li Jiatao Gu.
references.title: Cross-lingual retrieval for iterative selfsupervised training
references.venue: NeurIPS.
references.authors: Ferhan Ture Elizabeth Boschee.
references.title: Learning to translate for multilingual question answering
references.venue: EMNLP.
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault Remi Louf Morgan Funtowicz Joe Davison Sam Shleifer Patrick von Platen Clara Ma Yacine Jernite Julien Plu Canwen Xu Teven Le Scao Sylvain Gugger Mariama Drame Quentin Lhoest Alexander Rush
references.title: Transformers: State-of-the-art natural language processing
references.venue: In EMNLP (System Demonstrations)
references.authors: Lee Xiong Chenyan Xiong Ye Li Kwok-Fung Tang Jialin Liu Paul N. Bennett Junaid Ahmed Arnold Overwijk.
references.title: Approximate nearest neighbor negative contrastive learning for dense text retrieval
references.venue: ICLR.
references.authors: Linting Xue Noah Constant Adam Roberts Mihir Kale Rami Al-Rfou Aditya Siddhant Aditya Barua Colin Raffel.
references.title: mT5: A massively multilingual pre-trained text-to-text transformer
references.venue: NAACL.
sections.heading: 1 Introduction
sections.text: Multilingual open question answering (QA) is the task of answering a question from a large collection of multilingual documents. Most recent progress in open QA is made for English by building a pipeline based on a dense passage retriever trained on large-scale English QA datasets to find evidence passages in English (Lee et al., 2019; Karpukhin et al., 2020), followed by a reader that extracts an answer from retrieved passages. However, extending this approach to multilingual open QA poses new challenges. Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures (Asai et al., 2021; Lin et al., 2020). Nonetheless, large-scale cross-lingual open QA training data whose questions and evidence are in different languages are not available in many of those languages.
  To address these challenges, previous work in multilingual open QA (Ture and Boschee, 2016; Asai et al., 2021) translates questions into English, applies an English open QA system to answer in English, and then translates answers back to the target language. Those pipeline approaches suffer from error propagation of the machine translation component into the downstream QA, especially for low-resource languages. Moreover, they are not able to answer questions whose answers can be found in resources written in languages other than English or the target languages.
  In this paper, we introduce a unified many-to-many QA model that can answer questions in any target language by retrieving evidence from any language and generating answers in the target language. Our method (called CORA, Fig. 1) extends the retrieve-then-generate approach of English open QA (Lewis et al., 2020; Izacard and Grave, 2021b) with a single cross-lingual retriever and a generator that do not rely on language-specific retrievers or machine translation modules. The multilingual
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  retrieval module (mDPR) produces dense embeddings of a question and all multilingual passages, thereby retrieving passages across languages. The generation module (mGEN) is trained to output an answer in the target language conditioned on the retrieved multilingual passages. To overcome the aforementioned data scarcity issue, we automatically mine training data using external language links and train mDPR and mGEN iteratively. In particular, each iteration proceeds over two stages of updating model parameters with available training data and mining new training data cross-lingually by Wikipedia language links and predictions made by the models. This approach does not require any additional human annotations or machine translation, and can be applied to many new languages with low resources.
  Our experiments show that CORA advances the state of the art on two multilingual open QA datasets, XOR-TYDI QA (Asai et al., 2021) and MKQA (Longpre et al., 2020), across 26 typologically diverse languages; CORA achieves gains of 23.4 and 4.7 F1 points in XOR-TYDI QA and MKQA respectively, where MKQA data is not used for training. Moreover, CORA achieves F1 scores of roughly 30 over 8 languages on MKQA that have no training data or even reference Wikipedia documents, outperforming the state-of-the-art approach by 5.4 F1 points. Our controlled experiments and human analyses illustrate the impact of many-to-many cross-lingual retrieval in improving multilingual open QA performance. We further observe that through cross-lingual retrieval, CORA can find answers to 20% of the multilingual questions that are valid but are originally annotated as unanswerable by humans due to the lack of evidence in the English knowledge sources.
sections.heading: 2 Method
sections.text: We define multilingual open QA as the task of answering a question qL in a target language L given a collection of multilingual reference passages Cmulti, where evidence passages can be retrieved from any language. These passages come from Wikipedia articles that are not necessarily parallel over languages. We introduce CORA, which runs a retrieve-then-generate procedure to achieve this goal (Fig. 1). We further introduce a novel training scheme of iterative training with data mining (§ 2.2).
sections.heading: 2.1 CORA Inference
sections.text: CORA directly retrieves evidence passages from any language for questions asked in any target language, and then generates answers in the target language conditioned on those passages. More formally, the CORA inference consists of two steps of (i) retrieving passages Pmulti and (ii) generating an answer aL based on the retrieved passages. Pmulti can be in any language included in Cmulti.
  Pmulti = mDPR(qL,Cmulti), aL = mGEN(qL,Pmulti).
  Multilingual Dense Passage Retriever (mDPR). mDPR extends Dense Passage Retriever (DPR; Karpukhin et al., 2020) to a multilingual setting. mDPR uses an iterative training approach to fine-tune a pre-trained multilingual language model (e.g., mBERT; Devlin et al., 2019) to encode passages and questions separately. Once training is done, the representations for all passages from C
  multi are computed offline and stored locally. Formally, a passage encoding is obtained as follows: epL = mBERTp(p), where a passage p is a fixed-length sequence of tokens from multilingual documents. At inference, mDPR independently obtains a d-dimensional (d = 768) encoding of the
  question eqL = mBERTq(qL). It retrieves k passages with the k highest relevance scores to the question, where the relevance score between a passage p and a question qL is estimated by the inner product of their encoding vectors, ⟨eqL , ep⟩. Multilingual Answer Generator (mGEN). We use a multilingual sequence-to-sequence model (e.g., mT5; Xue et al., 2021) to generate answers in the target language token-by-token given the retrieved multilingual passages Pmulti. We choose a generation approach because it can generate an answer in the target language L from passages across different languages.1 Moreover, the generator can be adapted to unseen languages, some of which may have little or no translation training data. Specifically, the generator outputs the sequence probability for aL as follows:
  P (aL∣qL,Pmulti) = T
  ∏ i p(aLi ∣aL<i, qL,Pmulti), (1)
  where aLi denotes the i-th token in the answer, and T is the length of the answer. We append a language tag to the question to indicate the target language.
sections.heading: 2.2 CORA Training
sections.text: We introduce an iterative training approach that encourages cross-lingual retrieval and answer generation conditioned on multilingual passages (sketched in Fig. 2 and Alg. 1). Each iteration proceeds over two stages: parameter updates (§ 2.2.1) where mDPR and mGEN are trained on the current training data and cross-lingual data mining (§ 2.2.2) where training data are automatically expanded by Wikipedia language links and model predictions.
  Initial training data. The initial training data is a combination of multilingual QA datasets: XORTYDI QA and TYDI QA (Clark et al., 2020), and an English open QA dataset (Natural Questions, Kwiatkowski et al., 2019). Each training instance from these datasets comprises a question, a positive passage, and an answer. Note that annotations in the existing QA datasets have critical limitations: positive passages are taken either from English (Asai et al., 2021) or the question’s language (Clark et al., 2020). Further, most of the non-English languages are not covered. Indeed, when we only train mDPR on this initial set, it often learns to retrieve passages in the same languages or similar languages with irrelevant context or context without sufficient evidence to answer.
sections.heading: 2.2.1 Parameter Updates
sections.text: mDPR updates (line 3 in Alg. 1). Let D = {⟨qLi , p+i , p−i,1,⋯, p−i,n⟩}mi=1 be m training instances. Each instance consists of a question qLi , a passage that answers the question (positive passage) p + i , and n passages that do not answer the question (negative passages) p−i,j . For each question, we use positive passages for the other questions in the training batch as negative passages (in-batch negative, Gillick et al., 2019; Karpukhin et al., 2020). mDPR is updated by minimizing the negative log likelihood of positive passages:
  Lmdpr = − log exp(⟨eqLi , ep+i ⟩)
  exp(⟨eqLi , ep+i ⟩) +∑ n j=1 exp(⟨eqLi , ep−i,j ⟩)
  . (2)
  1An alternative approach of answer extraction requires translation for all language pairs (Asai et al., 2021).
  Algorithm 1: Iterative training that automatically mines training data. Data: Input QA pairs: (qL, aL)
  1 initialize training data B1 = (qL, aL, pgold),L = {Eng, L}; 2 while t < T do 3 Θ t mDPR ← Train(θt−1mDPR,Bt)/* Train mDPR */ 4 Pmulti ← mDPR(qL, embedding(Cmulti))/* Retrieve passages */ 5 θ t mGEN ← Train(θt−1mGEN , (qL, aL,Pmulti))/* Train mGEN */ 6 For L == Eng, Pmulti+ = LangLink(qL,Cmulti)) /* Mine data using Wikidata */ 7 For pi ∈ P
  multi: if mGEN(qL, pi) == aL then positives.add(pi) else negatives.add(pi) 8 B
  t+1 += (qL, aL, positives, negatives) /* Add new training data */ 9 t← t + 1
  10 end
  mGEN updates (lines 4-5 in Alg. 1). After updating mDPR, we use mDPR to retrieve top k passages Pmulti for each qL. Given these pairs of the question and the retrieved passages (qL,Pmulti) as input, mGEN is trained to generate answer aL autoregressively (Eq. (1)) and minimize the crossentropy loss. To train the model to generate in languages not covered by the original datasets, we translate aL to other languages using Wikipedia language links and create new synthetic answers.2 See Appendix § A.2 for more detail.
sections.heading: 2.2.2 Cross-lingual Data Mining
sections.text: After the parameter updates, we mine new training data using mDPR and Wikipedia language links and label the new data by mGEN predictions. This step is skipped in the final iteration.
  Mining by trained mDPR and language links (line 4, 6 in Alg. 1). Trained mDPR can discover positive passages in another language that is not covered by the initial training data. At each iteration, we use retrieved passages Pmulti for qL (line 4 in Alg. 1) as a source of new positive and negative passages. This enables expanding data between language pairs not in the original data.
  To cover even more diverse languages, we use language links and find passages in other languages that potentially include sufficient evidence to answer. Wikipedia maintains article-level language links that connect articles on the same entity over languages. We use these links to expand training data from the English QA dataset of Natural Questions (line 6 in Alg. 1). Denote a training instance by (qEn, aEn, pgold). We first translate the English answer aEn to a target language aL using language links. We use language links again to look up the English Wikipedia article that the gold passage pgold comes from. We then find articles in non-English languages in the reference documents C multi that correspond to this article. Although the language link-based automatic translation cannot handle non-entity answers (e.g., short phrases), this helps us to scale to new languages without additional human annotation or machine translation. We add all passages from these articles to Pmulti as positive passage candidates, which are then passed to mGEN to evaluate whether each of them leads to aL or not.
  Automatic labeling by mGEN predictions (lines 7-8 in Alg. 1). A passage pi from P multi may not always provide sufficient information to answer the question qL even when it includes the answer string aL. To filter out those spurious passages (Lin et al., 2018; Min et al., 2019), we take instances generated from the two mining methods described above, and run mGEN on each passage to predict an answer for the question. If the answer matches the correct answer aL, then the passage pi is labeled as a positive passage; otherwise we label the input passage as a negative passage. We assume that when mGEN fails to generate a correct answer given the passage, the passage may not provide sufficient evidence to answer; this helps us filter out spurious passages that accidentally contain an answer string yet do not provide any clue to answer. We add these new positive and negative passages to the training data, and in the next iteration, mDPR is trained on this expanded training set (§ 2.2.1).
  2This automatic answer translation is only done after the third epoch of initial training to prevent the model from overfitting to synthetic data.
sections.heading: 3 Experiments
sections.text: We evaluate CORA on two multilingual open QA datasets across 28 typologically diverse languages.3 CORA achieves state-of-the-art performance across 26 languages, and greatly outperforms previous approaches that use language-specific components such as question or answer translation.
sections.heading: 3.1 Datasets and Knowledge Sources
sections.text: Multilingual open QA datasets differ in covered languages, annotation schemes, and target application scenarios. We evaluate F1 and EM scores over the questions with answer annotations from two datasets, following the common evaluation practice in open QA (Lee et al., 2019).
  XOR-TYDI QA. XOR-TYDI QA (Asai et al., 2021) is a multilingual open QA dataset consisting of 7 typologically diverse languages, where questions are originally from TYDI QA (Clark et al., 2020) and posed by information-seeking native speakers. The answers are annotated by extracting spans from Wikipedia in the same language as the question (in-language data) or by translating English spans extracted from English Wikipedia to the target language (cross-lingual data). XOR-TYDI QA offers both training and evaluation data.
  MKQA. MKQA (Longpre et al., 2020) is an evaluation dataset created by translating 10k Natural Questions (Kwiatkowski et al., 2019) to 25 target languages. The parallel data enables us to compare the models’ performance across typologically diverse languages, in contrast to XOR-TYDI QA. MKQA has evaluation data only; XOR-TYDI QA and MKQA have five languages in common.
  Collection of multilingual documents Cmulti. We use the February 2019 Wikipedia dumps of 13 diverse languages from all XOR-TYDI QA languages and a subset of MKQA languages.4 We choose 13 languages to cover languages with a large number of Wikipedia articles and a variety of both Latin and non-Latin scripts. We extract plain text from Wikipedia articles using wikiextractor,5 and split each article into 100-token segments as in DPR (Karpukhin et al., 2020). We filter out disambiguation pages that distinguish pages that share the same article title6 as well as pages with fewer than 20 tokens, resulting in 43.6M passages. See more details in Appendix § B.2.
  Language categories. To better understand the model performance, we categorize the languages based on their availability during our training. We call the languages with human annotated gold paragraph and answer data seen languages. XOR-TYDI QA provides gold passages for 7 languages. For the languages in Cmulti without human-annotated passages, we mine new mDPR training data by our iterative approach (§ 2.2). We call these languages, which are seen during mDPR training, mDPR-seen. We also synthetically create mGEN training data as explained in § 2.2.1 by simply replacing answer entities with the corresponding ones in the target languages. The languages that are unseen by mDPR but are seen by mGEN are called mGEN-seen, and all other languages (i.e., included neither in mDPR nor mGEN training; 9 of the MKQA languages) unseen languages.
sections.heading: 3.2 Baselines and Experimental Setting
sections.text: We compare CORA with the following strong baselines adopted from Asai et al. (2021).
  Translate-test (MT + DPR). As used in most previous work (e.g., Asai et al., 2021), this method translates a question to English, extracts an answer in English using DPR, and then translates the answer back to the target language. The translation models are obtained from MarianMT (JunczysDowmunt et al., 2018) and trained on the OPUS-MT dataset (Tiedemann, 2012).
  Monolingual baseline (BM25). This baseline retrieves passages solely from the target language and extracts the answer from the retrieved passages. Training neural network models such as DPR is infeasible with a few thousands of training examples. Due to the lack of training data in most of
  3A full list of the language families and script types are in the appendix. 4Downloaded from https://archive.org/details/wikimediadownloads?and%5B%5D=
  year%3A%222019%22. 5https://github.com/attardi/wikiextractor 6https://en.wikipedia.org/wiki/Category:Disambiguation_pages.
  the target languages, we use a BM25-based lexical retriever implementation by Pyserini (Lin et al., 2021). We then feed the retrieved documents to a multilingual QA model to extract final answers.
  MT+Mono. This baseline combines results from the translate-test method and the monolingual method to retrieve passages in both English and the target language. Following Asai et al. (2021), we prioritize predictions from the monolingual pipeline if they are over a certain threshold tuned on XOR-TYDI QA development set; otherwise we output predictions from the translate-test method.7
  Closed-book baseline. This model uses an mT5-base8 sequence-to-sequence model that takes a question as input and generates an answer in the target language without any retrieval at inference time (Roberts et al., 2020). This baseline assesses the models’ ability to memorize and retrieve knowledge from its parameters without retrieving reference documents.
  CORA details. For all experiments, we use a single retriever (mDPR) and a single generator (mGEN) that use the same passage embeddings. mDPR uses multilingual BERT base uncased,9 and the generator fine-tunes mT5-base. We found that using other pre-trained language models such as mBART (Liu et al., 2020) for mGEN or XLM-R (Conneau et al., 2020) for mDPR did not improve performance and sometimes even hurt performance. We first fine-tune mDPR using gold passages from Natural Questions, and then further fine-tune it using XOR-TYDI QA and TYDI QA’s gold passage data. We exclude the training questions in Natural Questions and TYDI QA that were used to create the MKQA or XOR-TYDI QA evaluation set. We run two iterations of CORA training (§ 2.2) after the initial fine-tuning. All hyperparameters are in Appendix § B.5.
sections.heading: 4 Results and Analysis
sections.heading: 4.1 Multilingual Open QA Results
sections.text: XOR-TYDI QA. Table 1 reports the scores of CORA and the baselines in XOR-TYDI QA. CORA, which only uses a single retriever and a single generator, outperforms the baselines and the previous state-of-the-art model on XOR-TYDI QA by a large margin across all 7 languages. CORA achieves gains of 24.8 macro-averaged F1 points over the previous state-of-the-art method (GMT+GS), which uses external black-box APIs, and 23.4 points over the concurrent anonymous work (SER).
  MKQA. Tables 2 and 3 report the F1 scores of CORA and the baselines on over 6.7k MKQA questions with short answer annotations10 under seen and unseen settings. CORA significantly
  7For the languages not supported by Pyserini, we always output translate-test’s predictions. 8We did not use larger-sized variants due to our computational budget. 9The alternative of XLM-RoBERTa (Conneau et al., 2020) did not improve our results.
  10Following previous work in open QA but different from the official script of MKQA (Longpre et al., 2020), we disregard the questions labeled as “no answer”. As shown in our human analysis, it is difficult to prove an answer does not exist in the millions of multilingual documents even if the annotation says so.
  outperforms the baselines in all languages by a large margin except for Arabic and English. Note that Longpre et al. (2020) report results in a simplified setting with gold reference articles from the original Natural Questions dataset given in advance, and thus their results are not comparable. CORA yields larger improvements over the translate-test baseline in the languages that are distant from English and with limited training data such as Malay (Ms; 27.8 vs. 12.6) and Hebrew (He; 15.8 vs. 8.9). The performance drop of the translate-test model from English (43.3 F1) to other languages indicates the error propagation from the translation process. BM25 performs very poorly in some low-resource languages such as Thai because of the lack of answer content in the target languages’ Wikipedia. MT+Mono underpeforms the MT+DPR baseline in MKQA since it is challenging to rerank answers from two separate methods with uncaliberated confidence scores. In contrast, CORA retrieves passages across languages, achieving around 30 F1 on a majority of the 26 languages.
sections.heading: 4.2 Analysis
sections.text: Ablations: Impact of CORA components. We compare CORA with the following four variants to study the impact of different components. (i) mDPR1 + mGEN1 only trains CORA using the initial labeled, annotated data and measures the impact of the iterative training. (ii) DPR (trained NQ) + mGEN replaces mDPR with a multilingual BERT-based DPR trained on English data from Natural Questions (NQ), and encodes all passages in Cmulti. This configuration assesses the impact of cross-lingual training data. (iii) CORA, Cmulti={En} only retrieves from English during inference. This variant evaluates if English reference documents suffice to answer multilingual questions. (iv) mDPR+Ext.reader+MT replaces mGEN with an extractive reader model (Karpukhin et al., 2020) followed by answer translation. This variant quantifies the effectiveness of using a multilingual generation model over the approach that combines an extractive reader model with language-specific translation models. Note that for MKQA experiments, we sample the same 350 questions (∼5%) from the evaluation set for each language to reduce the computational cost over varying configurations.
  Results in Table 4 show performance drops in all variants. This supports the following claims: (i) the iterative learning and data mining process is useful, (ii) mDPR trained with cross-lingual data substantially outperforms DPR with multilingual BERT trained on monolingual data only, (iii) reference languages other than English are important in answering multilingual questions, and (iv) a multilingual generation model substantially boosts the model performance.
  Retrieval performance and relationship to the final QA performance. We evaluate CORA’s retrieval performance on MKQA using two recall metrics that measure the percentage of questions with at least one passage among the top 10 that includes a string in an answer set in the target language (RL@10) or in the union of answer sets from all languages that are available in MKQA (Rmulti@10). MKQA provides answer translations across 26 languages.
  Table 5 reports retrieval results for mDPR and multilingual BERT-based DPR trained on NQ: DPR (NQ). This is equivalent to (ii) from the ablations. We observe that mDPR performs well in IndoEuropean languages with Latin script, even when the language is unseen. Interestingly, there is a significant performance gap between RL@10 and Rmulti@10 in languages with non-Latin script (e.g., Japanese, Russian, Chinese); this suggests that our model often uses relevant passages from other languages with Latin script such as English or Spanish to answer questions in those languages with non-Latin script. Our mDPR outperforms DPR (NQ) by a large margin in unseen languages with limited resources, which are consistent with the findings in Table 3. Nevertheless, we still see low performance on Khmer and Thai even with the Rmulti@10 metric. We also observe that passage and query embeddings for those languages are far from other languages, which can be further studied in future work. We provide a two-dimensional visualization of the encoded passage representations in the appendix.
  Breakdown of reference languages. Fig. 3 breaks down retrieved reference languages for each target language. Our multilingual retrieval model often retrieves documents from the target language (if its reference documents are available), English, or its typologically similar languages. For example, mDPR often retrieves Spanish passages for Portuguese questions and Japanese passages for Chinese questions; while they are considered phylogenetically distant, Japanese and Chinese overlap in script.
  To further evaluate this, we conduct a controlled experiment: we remove Spanish, Swedish and Indonesian document embeddings and evaluate CORA on related languages: Danish, Portuguese and
  Malay. We observe performance drops of 1.0 in Danish, 0.6 in Portuguese, and 3.4 F1 points in Malay. This illustrates that while CORA allows for retrieval from any language in principle (many-to-many), cross-lingual retrieval from closer languages with more language resources is particularly helpful.
  Error analysis and qualitative examples. Table 6 analyzes errors from CORA by manually inspecting 50 Japanese and Spanish wrong predictions from MKQA. We observe six major error categories: (a) retrieval errors, (b) generating correct answers in a different language (different lang), (c) incorrect answer generation (incorrect answer), (d) answer annotation errors (e.g., a correct alias isn’t covered by gold answers, or Wikipedia information is inconsistent with English.), and (e) ambiguous or underspecified questions such as “who won X this year” (underspecified q). The table shows that both in Japanese and Spanish, the retrieval errors are dominant. In Japanese, CORA often generates correct answers in English, not in Japanese (different lang).
  Fig. 4 shows some qualitative examples. The first example shows an error in (b): mGEN is generating an answer in Russian, not in French though the answer itself is correct. This type of error happens especially when retrieved passages are in languages other than the target and English.
  Human evaluation on cross-lingual retrieval results. To observe how cross-lingual retrieval between distant languages is actually helping, we sample 25 Norwegian questions for which Spanish passages are included among the top 10 retrieved results. As seen in Fig. 3, CORA retrieves Spanish (es) passages for 6.8% of the Norwegian (no) questions. A Spanish speaker judges if the retrieved Spanish passages actually answer the given Norwegian questions.11 We found that in 96% of the cases, the retrieved Spanish passages are relevant in answering the question. One such example is presented in Fig. 4 (the second example).
  Human analysis on unanswerable questions. CORA retrieves passages from a larger multilingual document collection than the original human annotations. Thus, CORA may further improve the answer coverage over the original human annotations. MKQA includes questions that are marked as unanswerable by native English speakers given English knowledge sources. We sample 400 unanswerable Japanese questions whose top one retrieved passage is from a non-English Wikipedia article. Among these, 329 unanswerable questions are underspecified (also discussed in Asai and Choi, 2021). For 17 out of the 71 remaining questions, the answers predicted by CORA are correct. This finding indicates the significance of cross-lingual retrieval and potential room for improvement in annotating multilingual open QA datasets. The third example in Fig. 4 shows one of these cases.
sections.heading: 5 Related Work and Broader Impacts
sections.text: English and non-English open QA. Despite the rapid progress in open QA (Chen et al., 2017; Karpukhin et al., 2020), most prior work has been exclusively on English (Lewis et al., 2020; Izacard and Grave, 2021b). Several prior attempts to build multilingual open QA systems often rely on machine translation or language-specific retrieval models (Ture and Boschee, 2016; Asai et al., 2021).
  11During evaluation, we provide the original English questions from MKQA.
  Lewis et al. (2020) and Guu et al. (2020) introduce retrieve-then-generate approaches. Izacard and Grave (2021a) introduce an iterative training framework that uses attention weights from a generator model as a proxy for text relevance scores. Tran et al. (2020) introduce CRISS, a self-supervised pre-training approach consisting of a parallel sentence mining module and a sequence-to-sequence model, which are trained iteratively. Several recent work such as Xiong et al. (2021) improves DPR by mining and learning with hard examples. Our work is the first work that introduces a unified multilingual system for many-to-many open QA, which is a challenging task requiring massive-scale cross-lingual retrieval and has not been addressed in prior work. We introduce an iterative training and data mining approach guided by filtering from an answer generation model to automatically extend annotated data available only in high-resource languages to low-resource. This approach contributes to significant performance improvements in languages without annotated training data.
  Many-languages-one models. Several recent work introduces single multilingual models for many languages using pre-trained multilingual models such as mBERT or mT5 in many NLP tasks (e.g., entity linking: Botha et al., 2020; De Cao et al., 2021; semantic role labeling: Mulcaire et al., 2019b; Lyu et al., 2019; Fei et al., 2020; syntactic parsing: Mulcaire et al., 2019a; Kondratyuk and Straka, 2019). This work conducts the first large-scale study of a unified multilingual open QA model across many languages and achieves state-of-the-art performance in 26 typologically diverse languages.
  Synthetic data creation for machine reading comprehension. Alberti et al. (2019) introduce a method of generating synthetic machine reading comprehension data by automatically generating questions and filtering them out by a trained machine reading comprehension model. Several studies augment multilingual machine reading comprehension training data by generating new questionanswer pairs from randomly sampled non-English Wikipedia paragraphs (Riabi et al., 2021; Shakeri et al., 2020). This work focuses on multilingual open QA, which involves not only machine reading comprehension but also cross-lingual retrieval. A similar augmentation method for machine reading comprehension can be applied to further improve the answer generation component in CORA.
  Societal impacts. Our code and data are publicly available. CORA can perform open QA in unseen languages and can benefit society in building QA systems for low-resource languages, hence enabling research in that direction. Unlike previous models, CORA removes the necessity of external black-box APIs, and thus we can examine and address wrong answers due to model errors or misinformation present on Wikipedia. This would help us mitigate the potential negative impact from CORA or its subsequent models outputting a wrong answer when it is used by people who seek information.
sections.heading: 6 Conclusion
sections.text: To address the information needs of many non-English speakers, a QA system has to conduct crosslingual passage retrieval and answer generation. This work presents CORA, a unified multilingual many-to-many open QA model that retrieves multilingual passages in many different languages and generates answers in target languages. CORA does not require language-specific translation or retrieval components and can even answer questions in unseen, new languages. We conduct extensive experiments on two multilingual open QA datasets across 28 languages, 26 of which CORA advances the state of the art on, outperforming competitive models by up to 23 F1 points. Our extensive analysis and manual evaluation reveal that CORA effectively retrieves semantically relevant passages beyond language boundaries, and can even find answers to the questions that were previously considered unanswerable due to lack of sufficient evidence in annotation languages (e.g., English). Nonetheless, our experimental results show that the retrieval component still struggles to find relevant passages for queries in some unseen languages. Our analysis also showed that CORA sometimes fails to generate an answer in the target language. In future work, we aim to address these issues to further improve the performance and scale our framework to even more languages.
sections.heading: Acknowledgement
sections.text: This research was supported by NSF IIS-2044660, ONR N00014-18-1-2826, gifts from Google, the Allen Distinguished Investigator Award, the Sloan Fellowship, and the Nakajima Foundation Fellowship. We thank anonymous reviewers, area chairs, Eunsol Choi, Sewon Min, David Wadden, and the members of the UW NLP group for their insightful feedback on this paper, and Gabriel Ilharco for his help on human analysis.
title: One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval