Recent advances in open-domain QA have led to strong models based on dense retrieval, but only focused on retrieving textual passages. In this work, we tackle open-domain QA over tables for the first time, and show that retrieval can be improved by a retriever designed to handle tabular context. We present an effective pre-training procedure for our retriever and improve retrieval quality with mined hard negatives. As relevant datasets are missing, we extract a subset of NATURALQUESTIONS (Kwiatkowski et al., 2019) into a Table QA dataset. We find that our retriever improves retrieval results from 72 . 0 to 81 . 1 recall@10 and end-to-end QA results from 33 . 8 to 37 . 7 exact match, over a BERT based retriever. Models for question answering (QA) over tables usually assume that the relevant table is given during test time. This applies for semantic parsing (e.g., for models trained on SPIDER (Yu et al., 2018)) and for end-to-end QA (Neelakantan et al., 2016; Herzig et al., 2020). While this assumption simplifies the QA model, it is not realistic for many use-cases where the question is asked through some open-domain natural language interface, such as web search or a virtual assistant. In these open-domain settings, the user has some information need, and the corresponding answer resides in some table in a large corpus of tables. The QA model then needs to utilize the corpus as an information source, efficiently search for the relevant table within, parse it, and extract the answer. Recently, much work has explored open-domain QA over a corpus of textual passages (Chen et al., 2017; Sun et al., 2018; Yang et al., 2019; Lee et al., 2019, inter alia ). These approaches usually follow a two-stage framework: (1) a retriever first selects a small subset of candidate passages relevant to the  Work completed while interning at Google. question, and then (2) a machine reader examines the retrieved passages and selects the correct answer. While these approaches work well on free text, it is not clear whether they can be directly applied to tables, as tables are semi-structured, and thus different than free text. In this paper we describe the first study to tackle open-domain QA over tables, and focus on modifying the retriever. We follow the two-step approach of a retriever model that retrieves a small set of candidate tables from a corpus, followed by a QA model (Figure 1). Specifically, we utilize dense retrieval approaches targeted for retrieving passages (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020), and modify the retriever to better handle tabular contexts. We present a simple and effective pre-training procedure for our retriever, and further improve its performance by mining hard negatives using the retriever model. Finally, as relevant open domain datasets are missing, we process NATURALQUESTIONS (Kwiatkowski et al., 2019) and extract 11K examples where the answer resides in some table. Our model and data generation code as well as the pre-trained model are publicly available at https://github.com/google-research/ tapas . We formally define open domain extractive QA over tables as follows. We are given a training set of N examples D train = { ( q i , T i , a i ) } Ni =1 , where q i is a question, T i is a table where the answer a i resides, and a corpus of M tables C = { T i } Mi =1 . The answer a i is comprised of one or more spans of tokens in T i . Our goal is to learn a model that given a new question q and the corpus C returns the correct answer a . Our task shares similarities with open domain QA over documents (Chen et al., 2017; Yang et al., 2019; Lee et al., 2019), where the corpus C consists of textual passages extracted from documents instead of tables, and the answer is a span that appears in some passage in the corpus. As in these works, dealing with a large corpus (of tables in our setting), requires relevant context retrieval. Naively applying a QA model, for example TAPAS (Herzig et al., 2020), over each table in the large corpus is not practical because inference is too expensive. To this end we break our system into two in-dependent steps. First, an efficient table retriever component selects a small set of candidate tables CR from a large corpus of tables C . Second, we apply a QA model to extract the answer a given the question q and the candidate tables CR . In this section we describe our dense table retriever (DTR), which retrieves a small set of K candidate tables CR given a question q and a corpus C . In this work we set K = 10 and take C to be the set of all tables in the dataset we experiment with (see 6). As in recent work for open domain QA on passages (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020; Chen et al., 2021; Oguz et al., 2020), we also follow a dense retrieval architecture. As tables that contain the answer to q do not necessarily include tokens from q , a dense encoding can better capture similarities between table contents and a question. For training DTR, we leverage both in-domain training data D train , and automatically constructed pre-training data D pt of text-table pairs (see below). meaningful way, by capturing their specific structure. Traditional information retrieval methods such as BM25 are targeted to capture token overlaps between a query and a textual document, and other dense encoders are pre-trained language models (such as BERT) targeted for text representations. Recently, Herzig et al. (2020) proposed TAPAS, an encoder based on BERT, designed to contextually represent text and a table jointly. TAPAS includes table specific embeddings that capture its structure, such as row and column ids. In DTR, we use TAPAS to represent both the query q and the table T . For efficient retrieval during inference we use two different TAPAS instances (for q and for T ), and learn a similarity metric between them as Lee et al. (2019); Karpukhin et al. (2020). More concretely, the TAPAS encoder TAPAS ( x 1 , [ x 2 ]) takes one or two inputs as arguments, where x 1 is a string and x 2 is a flattened table. We then define the retrieval score as the inner product of dense vector representations of the question q and the table T : h q = W q TAPAS q ( q ) [CLS] h T = WTTAPAST ( title ( T ) , T ) [CLS] S ret ( q, T ) = h Tq h T , where TAPAS (  ) [CLS] returns the hidden state for the CLS token, W q and WT are matrices that project the TAPAS output into d = 256 dimensional vectors, and title ( T ) is the page title for table T . We found the table's page title to assist in retrieving relevant tables, which is also useful for Wikipedia passage retrieval (Lee et al., 2019). Training The goal of the retriever is to create a vector space such that relevant pairs of questions and tables will have smaller distance (which results in a large dot product) than the irrelevant pairs, by learning an embedding. To increase the likelihood of gold ( q, T ) pairs, we train the retriever with in-batch negatives (Gillick et al., 2019; Henderson et al., 2017; Karpukhin et al., 2020). Let { ( q i , T i ) } Bi =1 be a batch of B examples from D train , where for each q i , T i is the gold table to retrieve, and for each j (cid:54) = i we treat T j as a negative. We now define the likelihood of the gold table T i as: p ( T i | q i ) = exp[ S ret ( q i , T i )] (cid:80) Bj =1 exp[ S ret ( q i , T j )] . questions and tables respectively. Then, S = QTT gives an B  B matrix where the logits from the gold table are on the diagonal. We then train using a row-wise cross entropy loss where the labels are a B  B identity matrix. Pre-training One could train our retriever from scratch, solely relying on a sufficiently large in-domain training dataset D train . However, we find performance to improve after using a simple pretraining method for our retriever. Lee et al. (2019) suggest to pre-train a textual dense retriever using an Inverse Cloze Task (ICT). In ICT, the goal is to predict a context given a sentence s . The context is a passage that originally contains s , but with s masked. The motivation is that the relevant context should be semantically similar to s , and should contain information missing from s . Similarly, we posit that a table T that appears in close proximity to some text span s is more relevant to s than a random table. To construct a set D pt = { ( s i , T i ) } Mi =1 that consists of M pretraining pairs ( s, T ) , we use the pre-training data from Herzig et al. (2020). They extracted text-table pairs from 6.2M Wikipedia tables, where text spans were sampled from the table caption, page title, page description, segment title and text of the segment the table occurs in. This resulted in a total of 21.3M text-table ( s, T ) pairs. While Herzig et al. (2020) uses extracted ( s, T ) pairs for pretraining TAPAS with a masked language modeling objective, we pre-train DTR from these pairs, with the same objective used for in-domain data. Hard Negatives Following similar work (Gillick et al., 2019; Karpukhin et al., 2020; Xiong et al., 2021), we use an initial retrieval model to extract the most similar tables from C for each question in the training set. From this list we discard each table that does contain the reference answer to remove false negatives. We use the highest scoring remaining table as a particular hard negative. Given the new triplets of question, reference table and mined negative table, we train a new model using a modified version of the in-batch negative training discussed above. Given Q and S as defined above and a new matrix N ( B  d ) that holds the representations of the negative tables, S (cid:48) = QNT gives another B  B matrix that we want to be small in value (possibly negative). If we concatenate S and S (cid:48) row-wise we get a new matrix for which we can perform the same cross entropy training as before. The label matrix is now obtained by concatenating an identity matrix row-wise with a zero matrix. Inference During inference time, we apply the table encoder TAPAST to all the tables T  C offline. Given a test question q , we derive its representation h q and retrieve the top K tables with representations closest to h q . In our experiments, we use exhaustive search to find the top K tables, but to scale to large corpora, fast maximum inner product search using existing tools such as FAISS (Johnson et al., 2019) and SCANN (Guo et al., 2020) could be used, instead. A reader model is used to extract the answer a given the question q and K candidate tables. The model scores each candidate and at the same time extracts a suitable answer span from the table. Each table and question are jointly encoded using a TAPAS model. The score is a simple logistic loss based on the CLS token, as in Eisenschlos et al. (2020). The answer span extraction is modeled as a soft-max over all possible spans up to a certain length. Spans that are located outside of a table cell or that cross a cell are masked. Following Lee et al. (2017, 2019), the span representation is the concatenation of the contextual representation of the first and last token in the span s : h start = TAPAS r ( q, title ( T ) , T )[ START ( s )] h end = TAPAS r ( q, title ( T ) , T )[ END ( s )] S read ( q, T ) = MLP ([ h start , h end ]) . The training and test data are created by running a retrieval model. We extract the K = 10 highest scoring candidate tables for each question. At training time we add the reference table if it is missing from the candidates. At inference time all table candidates are processed and the answer of the candidate with the highest score is returned as the predicted answer. We create a new English dataset called NQ-TABLES from NATURALQUESTIONS (Kwiatkowski et al., 2019) (NQ). Concurrently with this work, Zayats et al. (2021) study a similar subset of NQ but without the retrieval aspect. from real Google search queries and the answers are spans in Wikipedia articles identified by annotators. Although the answers for most questions appear in textual passages, we identified 12K examples where the answer resides in a table, and can be used as a QA over tables example. To this end, we form NQ-TABLES that consists of ( q, T, a ) triplets from these examples. Tables are extracted from the article's HTML, and are normalized by transposing infobox tables. We randomly split the original NQ train set into train and dev (based on a hash of the page title) and use all questions from the original NQ dev set as our test set. To construct the corpus C , we extract all tables that appear in articles in all NQ sets. NQ can contain the same Wikipedia page in different versions which leads to many almost identical tables. We merge close duplicates using the following procedure. For all tables that occur on the same Wikipedia page we flatten the entire table content, tokenize it and compute l 2 normalized uni-gram vectors of the token counts of each table. We then compute the pair-wise cosine similarity of all tables. We iterate over the table pairs in decreasing order of similarity and attempt to merge them into clusters. This is essentially a version of single link clustering. In particular, we will merge two tables if the similarity is > 0 . 91 , they do not occur on the same version of the page, their difference is rows is at most 2 and they have the same number of columns. Dataset sizes are given in the following table: train dev test corpus C 9,594 1,068 966 169,898 Retriever Reader EM F1 Oracle EM Oracle F1 BM25 TAPAS 21.46 28.24 29.51 40.79 DTR-Text BERT 29.58 37.38 39.39 51.48 DTR-Text TAPAS 33.78 43.49 42.83 56.46 DTR-Schema TAPAS 32.75 42.19 42.63 55.05 DTR TAPAS 35.50 45.44 46.09 59.01 DTR +hnbm25 TAPAS 36.61 46.74 47.46 60.72 DTR +hn TAPAS 37.69 47.70 48.20 61.50 Table 2: QA results on NQ-TABLES test set. Details about the experimental setup are given Appendix A. Retrieval Baselines We consider the following baselines as alternatives to DTR. We use the BM25 (Robertson and Zaragoza, 2009) implementation of Gensim ( Rehurek and Sojka, 2010) 1 . To measure if a table-specific encoder is necessary, we implement DTR-TEXT , where the retriever is initialized from BERT (Devlin et al., 2019) instead of TAPAS. To test whether the content of the table is relevant, we experiment with DTR-SCHEMA , where only the headers and title are used to represent tables. Retrieval Results Table 1 shows the test results for table retrieval (dev results are in Appendix B). We report recall at K (R@K) metrics as the fraction of questions for which the highest scoring K tables contain the reference table. We find that all dense models that have been pre-trained out-peform the BM25 baseline by a large margin. The model that uses the TAPAS table embeddings (DTR) out-performs the dense baselines by more than 1 point in R@10. The addition of mined negatives (DTR +hn) yields an additional improvement of more than 5 points. Mining negatives from DTR works better than mining negatives from BM25 (DTR +hnbm25, +0.6 R@10). End-to-End QA Results for end-to-end QA experiments are shown in Table 2 (dev results are in Appendix B). We use the exact match (EM) and token F1 metrics as implemented in SQUAD (Ra-jpurkar et al., 2016). 2 We additionally report oracle 1 We find that recall improves if the document title and table header tokens are counted multiple times. In all experiments we use a count of 15. 2 https://worksheets. We again find that all dense models out-perform the BM25 baseline. A TAPAS-based reader outperforms a BERT reader by more than 3 points in EM. The simple DTR model out-performs the baselines by more than 1 point in EM. Hard negatives from BM25 (+hnbm25) improve DTR's performance by 1 point, while hard negatives from DTR (+hn) improve performance by 2 points. We additionally perform a McNemar's significance test for our proposed model, DTR+hn, and find that it performs significantly better (p<0.05) than all baselines. Analysis Analyzing the best model in Table 2 (DTR +hn) on the dev set, we find that 29% of the questions are answered correctly, 14% require a list answer (which is out of scope for this paper), 12% do not have any table candidate that contains the answer, for 11% the model does not select a table that contains the answer, and for 34% the reader fails to extract the correct span. We further analyzed the last category by manually annotating 100 random examples. We find that for 23 examples the answer is partially correct (usually caused by inconsistent span annotations in NQ). For 11 examples the answer is ambiguous (e.g., the release date of a movie released in different regions). For 22 examples the table is missing context or does only contain the answer accidentally. Finally, 44 examples are wrong, usually because they require some kind of table reasoning, like computing the maximum over a column, or using common sense knowledge. In this paper we demonstrated that a retriever designed to handle tabular context can outperform other textual retrievers for open-domain QA on tables. We additionally showed that our retriever can be effectively pre-trained and improved by hard negatives. In future work we aim to tackle multimodal open-domain QA, combining passages and tables as context. We would like to thank William Cohen, Sewon Min, Yasemin Altun and the anonymous reviewers for their constructive feedback, useful comments and suggestions. This work was completed in partial fulfillment for the PhD degree of the first author, which was also supported by a Google PhD fellowship.
abstain objective result method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method result method other method method abstain method method method abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain method method method abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain other abstain method method abstain abstain method abstain method method abstain method method abstain method method method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain other result abstain abstain abstain abstain result abstain result abstain abstain abstain objective result objective other other