The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge. Although prior work has sought to increase the knowledge coverage by incorporating structured knowledge beyond text, accessing heterogeneous knowledge sources through a unified interface remains an open question. While data-to-text generation has the potential to serve as a universal interface for data and text, its feasibility for downstream tasks remains largely unknown. In this work, we bridge this gap and use the data-to-text method as a means for encoding structured knowledge for ODQA. Specifically, we propose a verbalizer-retriever-reader framework for ODQA over data and text where verbalized tables from Wikipedia and graphs from Wikidata are used as augmented knowledge sources. We show that our U nified D ata and T ext QA , UDT-QA , can effectively benefit from the expanded knowledge index, leading to large gains over text-only baselines. Notably, our approach sets the single-model state-of-the-art on Natural Questions. Furthermore, our analyses indicate that verbalized knowledge is preferred for answer reasoning for both adapted and hot-swap settings. Pretrained language models (Devlin et al., 2019; Brown et al., 2020) have been shown to store certain knowledge (linguistic or factual) implicitly in parameters (Manning et al., 2020; Petroni et al., 2019; Roberts et al., 2020), partially explaining the superior generalization abilities over downstream tasks. However, besides the well-known hallucination issue, the implicit knowledge learned through language modeling objective over text struggles at reflecting up-to-date knowledge from text and structured data for answering open-domain questions. To overcome this, recent work on open domain question answering (ODQA) focuses on the semi-parametric method (Karpukhin et al., 2020; Guu et al., 2020) where the pretrained language models can leverage external explicit knowledge sources for reasoning. For example, in the retriever-reader framework (Min et al., 2021, inter alia ), the reader produces answers by grounding on the relevant evidence from the retriever, the interface to the explicit knowledge source (Wikipedia text passages). In this work, we focus on the semi-parametric approach for ODQA going beyond textual knowledge. Specifically, we are interested in the question: Can we develop a viable unified interface over a realistic heterogeneous knowledge source containing both data and text? Recent retriever-reader models (Oguz et al., 2020; Agarwal et al., 2021) have demonstrated that expanding the textual knowledge source with more structured data is beneficial. However, only knowledge base (KB) is considered in (Agarwal et al., 2021), limiting the applicability of their method to other structured data. In (Oguz et al., 2020), both tables and KB triples are simply linearized as inputs to the reader, but different retrievers are required for individual cases. Here, we propose a verbalizer-retriever-reader semi-parametric framework, UDT-QA , which provides a unification of both representation and model for ODQA over data and text. The key idea is to augment the retriever with a data-to-text verbalizer for accessing heterogeneous knowledge sources, i.e. KB graphs from WikiData, tables and passages from Wikipedia. Given its potential in providing a universal interface for data and text, data-to-text generation is increasingly popular (Gardent et al., 2017; Parikh et al., 2020; Nan et al., 2021) with various methods developed recently for converting structured knowledge into natural language (Wang et al., 2020; Ribeiro et al., 2020; Chen et al., 2020b). Nevertheless, most existing work has focused on intrinsic 1605 evaluations exclusively, i.e. the quality of generated text measured by metrics like BLEU (Papineni et al., 2002), leaving its usefulness on downstream tasks largely unknown. Moreover, it remains unclear whether a single data-to-text model is able to verbalize heterogeneous structured data effectively. To bridge the gap, we develop a novel data-to-text generation paradigm for our framework. We introduce data filtering and beam selection to maximize the faithful coverage of the input information. To remedy the lack of in-domain data, we further propose an iterative training approach to augment the existing data-to-text training set with high quality outputs selected from the target domain. With this verbalizer, we convert all tables from Wikipedia (10x more than (Oguz et al., 2020)) and sub-graphs from Wikidata together with Wikipedia text passages as the knowledge source for ODQA. We first validate our data-to-text method using intrinsic metrics on DART (Nan et al., 2021) and additional faithfulness evaluation on the target ODQA data. We show that our data-to-text approach can effectively improve the target-domain faithful metric without compromising too much on the intrinsic metrics. To further evaluate the end-to-end effectiveness, we experiment with UDT-QA on the ODQA task using a recent state-of-the-art (SOTA) retriever-reader pipeline, including DPR (Karpukhin et al., 2020) and UnitedQA (Cheng et al., 2021b). Consistent with previous work, our results also suggest that extra knowledge source is beneficial for ODQA. Notably, we find that the verbalized knowledge is favored by the reader compared to the raw format (linearization), especially when the structured data size is comparable to text, leading to more pronounced improvements. Overall, UDT-QA shows large improvements over text-only baselines and performs competitively with more complicated methods on both Natural Questions (NQ) (Kwiatkowski et al., 2019) and WebQuestions (WebQ) (Berant et al., 2013). In particular, UDT-QA achieves new SOTA on NQ under the single-model open-book setting. 1 2 Overview of UDT-QA In this section, we present the overall pipeline of our UDT-QA framework for ODQA over data and text (Figure 1). The major difference between our approach and the popular retriever-reader ODQA 1 Data and code available at https://github.com/ Mayer123/UDT-QA systems (Min et al., 2021, inter alia ) is the use of a data-to-text verbalizer (3) for converting structured data into natural language text, i.e. virtual documents, as the universal knowledge source. Here, we consider two types of structured knowledge (4.2)  tables and KB sub-graphs. After verbalizing the structured knowledge, a subsequent pipeline consisting of a DPR retriever and a UnitedQA-E reader is used for answer inference. Since the retriever and reader are not the main focus of this work, we only briefly describe them below. The DPR retriever (Karpukhin et al., 2020) is a bi-encoder model consisting of a question encoder and a context encoder, which is used for data and text retrieval. Following previous work (Karpukhin et al., 2020; Oguz et al., 2020), we use the uncased BERT-base (Devlin et al., 2019) model as the encoder, where the [CLS] token representation is used as the document/question vector. During training, positive and negative pairs of (question, context) are used to update the model. For inference, the entire document index is encoded with context encoder and the encoded question vector is used to retrieve the top documents with highest dot-product scores. The UnitedQA-E (Cheng et al., 2021b) is an extractive reader based on ELECTRA (Clark et al., 2020) trained with enhanced objectives (Cheng et al., 2021a, 2020) for answer inference. Here, a pair of a question and a support passage is jointly encoded into neural text representations. These representations are used to compute scores of possible answer begin and end positions, which are then used to compute probabilities over possible answer spans. Finally, the answer string probabilities are computed based on the aggregation over all possible answer spans from the entire set of support passages. Here, we formally describe the data-to-text model developed in this paper, including the input format (3.1) and the adaptation for ODQA (3.2). Given a structured data input D , the data-to-text generator G aims to generate a natural language passage P that faithfully describes the information presented in D . In the literature, the structured data input can be in the form of a set of triples (Nan et al., 2021), a few highlighted cells from 1606 Figure 1: An overview of UDT-QA based on the verbalizer-retriever-reader pipeline. a table (Parikh et al., 2020) or a full table (Chen et al., 2020a). Correspondingly, P could a simple surface-form verbalization of D ( e.g. when D is a triple set) or a high-level summarization in case of a full table or a large KB graph. Since we consider (noisy) tables/KB sub-graphs of arbitrary size in this paper, directly feeding the entire input into the generator is not feasible, likely incurring signifi-cant computation challenges. Moreover, it is also desirable to maximize the information coverage of P so that most relevant information in D can be leveraged by the downstream QA retriever and reader. Based on this, we verbalize both tables and KB graphs at a fine-grained level. In this work, we verbalize tables row by row, i.e. input each table row to G individually, where each row is a set of cells r = { c i } ki =1 , and k is the number of cells in the corresponding row. Most relevant to our setting, recent work (Nan et al., 2021) represents each cell in a triple. To form such triples, they manually annotate the tree ontology of column headers and then create triples using table title, headers, cell value and header relations, e.g. ( [TABLECONTEXT], [title], LeBron James ), ( LeBron James, League, NBA ) where LeBron James is the parent cell. Although such triples with fine-grained ordering may help guide the generator, directly applying such a generator to a target domain with no ontology annotation (our case) likely results in degradation. To overcome this, we propose to convert the triple set to pairs, e.g. ( [title], LeBron James ), ( League, NBA ). We find such conversion has little impact on the intrinsic evaluation (5). After all rows are verbalized, we assemble the text outputs back to form the verbalized table. For KB, we follow previous work (Agarwal et al., 2021) and break the KB into small sub-graphs based on subject entity. Here, each sub-graph contains one central entity and its neighbors. Although this conversion would inevitably create undesirable artifacts ( e.g. hurdles for multi-hop reasoning across sub-graphs), this preprocessing allows us to unify the input representations for both table and KB graphs, making it possible for a single verbalizer to convert structured knowledge into text format. Specifically, we convert all KB sub-graphs into the same format as table cell sets above, where the subject entity is treated as the title and all the edges are represented using pairs in the form of ( relation , object ). Then we verbalize each sub-graph with the generator G . Examples of input and output for table rows and KB sub-graphs are shown in Figure 1. 3.2 Improved Data-to-Text Model Training A known problem in data-to-text generation is that the model tends to hallucinate or neglect information in the input (Wang et al., 2020; Agarwal et al., 2021). Faithfulness and information coverage is especially important when we apply the verbalized output to knowledge-intensive downstream tasks like ODQA. To address this, we subsample training data T such that the instances are filtered out if they are likely to steer model towards missing information. In particular, we compute ROUGE-1 (Lin, 2004) scores between the input and target of training instances and filter out those whose scores are below a certain threshold. We denote the filtered version as T-F . Examples of the filtered instances can be found in Table 11, as we discuss more in Appendix F, these instances may bias the model towards unwanted behaviors. 1607 Another challenge we face is that most data-to-text training examples have succinct structured inputs. In other words, the cells in the structured input are usually single words or short phrases with corresponding short target sentences as well. In our case, a number of tables contain large cells with dozens of words. Models trained with existing data likely have a hard time verbalizing such inputs faithfully. To alleviate this domain-mismatch issue, we propose an iterative training set-up. In the first iteration, we train a generator on T-F . Then we apply the generator to our data. We then find high quality verbalized outputs based on the ROUGE-1 score between the model inputs and model outputs, and sample instances with score higher than a threshold for the next-round training. We sample instances up to the same size of T-F , and denote this set as ID-T (examples shown in Table 11). Finally, we mix the ID-T with T-F and train a second generator for verbalization. Following recent work (Nan et al., 2021), we use the pretrained T5-Large (Raffel et al., 2020) model as our generator. Given paired training examples consisting of a structured data input and a target sentence, we finetune the T5 model to maximize the log-likelihood of generating the corresponding target sentences. Here, we follow the same experimental setup as (Ribeiro et al., 2020). In this section, we describe the data used for experiments and sources of structured knowledge. In this paper, we use DART (Nan et al., 2021) to train our verbalizer (data-to-text) and two ODQA datasets, NQ and WebQ, to train and evaluate our pipeline, with the same split as in (Lee et al., 2019) provided by (Karpukhin et al., 2020). Below we provide a brief description of each dataset and refer readers to their papers for details. DART is a data-to-text dataset containing pairs of (triple-set, sentences) collected from WebNLG (Gardent et al., 2017), E2E (Novikova et al., 2017) and crowdsourcing based on tables found in Wik-iSQL (Zhong et al., 2017) and WikiTableQuestions (Pasupat and Liang, 2015). We collect knowledge-answerable questions from NQ and WebQ in order to evaluate our verbalizer and construct the retrieval training data. Specifically, we find questions in the original NQ training set that can be answered by a table. For each question, we search through tables in its associated HTML page to locate exact answer matches. In total, we collected 14,164 triples of (question, answer, gold table) from NQ train and dev sets as NQ-table-Q . On WebQ, we find questions that can be answered by KB via expanding from question entities and search for their 1-hop neighbors. If an answer entity is matched, we keep this sub-graph. In total, we collected 2,397 triples of (question, answer, sub-graph) from WebQ train and dev set as WebQ-KB-Q . In addition to regular Wikipedia text passages, we consider two types of structured knowledge  tables tables from Wikipedia and KB graphs from Wikidata. For tables from Wikipedia, we follow OTT-QA (Chen et al., 2021b) with slight modifica-tions. Chen et al. (2021b) only consider tables in good format, i.e. tables with no empty cell, multi-column or multi-row, and restrict the tables to have at most 20 rows or columns. Instead, we remove such constraints and keep everything with the <table> tag, resulting in a larger and noisier table set. We denote this more realistic set of tables as OTT-tables . Note Oguz et al. (2020) only consider tables from the original NQ HTMLs. In addition to the size difference, OTT-tables are crawled from a more recent Wikipedia dump than the NQ version. To study the impact of knowledge source size, we also process tables from the NQ HTML pages with the heuristic suggested by (Herzig et al., 2021) to de-duplicate tables and filter lengthy cells (>80 words). We denote this set of tables as NQ-tables . To avoid overlap, we remove tables from OTT-tables whose page title are in NQ-tables set. In total, we have a All-tables set with 2.2M tables from OTT-tables and 210K tables from NQ-tables , respectively. For KB graphs, we consider using the English Wikidata (Vrandecic and Krtzsch, 2014) as our KB due to its broad coverage and high quality, not-1608 Intrinsic Eval Extrinsic Eval Training Set # Examples BLEU METEOR TER MoverScore BERTScore BLEURT Ans Cov DART (Nan et al., 2021) 62,659 50.66 0.40 0.43 0.54 0.95 0.44 DART ours ( T ) 62,628 51.05 0.40 0.43 0.54 0.95 0.43 95.4 DART ( T-F ) 55,115 51.04 0.41 0.43 0.54 0.95 0.43 96.0 DART ( T-F + ID-T ) 110,230 50.59 0.41 0.44 0.54 0.95 0.43 98.4 Table 1: Intrinsic and extrinsic evaluations of verbalization approaches on DART test and NQ-table-Q (4.1), respectively. ing its predecessor Freebase is no longer maintained despite its popularity in research. In order to be comparable with recent work (Agarwal et al., 2021), we directly use their partitioned KB graphs from WikiData in our experiments, which is denoted as WD-graphs . In this section, we evaluate our data-to-text model with both intrinsic and extrinsic metrics. Since intrinsic metrics are probably less correlated with the downstream performance, we use them only as a sanity check for generation quality and focus on using an extrinsic metric for selecting models. Intrinsic Evaluation : Since our model is developed mainly on DART, we first conduct the intrinsic evaluation on the DART test set to measure the impact of our improved data-to-text methods, i.e. data filtering and iterative training. Following (Nan et al., 2021), we use the official evaluation metrics including BLEU, METEOR (Banerjee and Lavie, 2005), TER, MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2020) and BLEURT (Sel-lam et al., 2020). Table 1 summarizes different data-to-text models on DART test. As we can see, the resulting model trained with our data conversion (row 2) performs on par with the model using the original format (row 1). More interestingly, filtering short samples has almost no impact on the verbalizer performance (row 3). Lastly, iterative training with additional target domain data (row 4) slightly hurts on BLEU and TER and achieves similar performances on other metrics. Overall, our verbalizer with the proposed data conversion and improved training remains very effective on DART. Extrinsic Evaluation : Since we are interested in applying verbalized knowledge for ODQA, the QA model is more likely to predict the correct answer only if the answer still exists after the verbalization. Therefore, we also evaluate each generator using a metric more related with the downstream task performance: answer coverage . Specifically, we compute the answer coverage as the percentage of examples that the answer present in the raw structured knowledge is still preserved in the corresponding verbalized output. First, we compute the answer coverage of different generators discussed in the previous section on NQ-table-Q where tables are known to contain question-triggering content. The scores are reported in the last column of Table 1. Due to more lengthy tables in NQ-table-Q , data filtering improves the answer coverage as expected. Moreover, model trained with our iterative training demonstrates substantial improvements in answer coverage, indicating that our approach is highly effective for converting tables into text. Examples for comparing different verbalizer outputs are shown in Table 12 in Appendix F. Later, we use this best generator to verbalize All-tables . We use beam search of size 10 and save all beams. To retain as much input information as possible, a re-ranking stage is carried out over these predictions based on the ROUGE-1 score between the model inputs and model outputs. The highest ranked prediction is then used as the final output. Lastly, we directly apply our best generator (DART T-F + ID-T) for verbalizing KB graphs. To evaluate the performance, we compare our model with the recent method KELM-verbalizer (Agar-wal et al., 2021) using answer coverage on the set WebQ-KB-Q where KB sub-graphs are known to contain answer entities. Although never tuned for KB graph inputs, our model achieves 99.6 on answer coverage, outperforming the KELM-verbalizer (97.8 on answer coverage) by a large margin. This suggests that our data-to-text approach is highly effective for both tables and KB sub-graphs. Here we present our main experiments on ODQA over data and text. For regular Wikipedia text, we use the same index containing 21M passages as 1609 Model NQ WebQ Without Structured Knowledge DPR (Karpukhin et al., 2020) 41.5 35.2 UnitedQA (Cheng et al., 2021b) 51.8 48.0 With Structured Knowledge KEALM (Agarwal et al., 2021) 41.5 43.9 UnitK-QA (Oguz et al., 2020) 54.1 57.8 UDT-QA w/ Raw Single Data 54.7 51.4 UDT-QA w/ Verbalized Single Data 55.2 52.0 UDT-QA w/ Verbalized Hybrid Data 55.1 52.5 Table 2: End-to-end open-domain QA evaluation of UDT-QA in comparison to recent state-of-the-art models on the test sets of NQ and WebQ. in (Karpukhin et al., 2020). To augment text, two settings are considered, i.e. the single data setting and the hybrid data setting. In the single data setting for NQ, we augment the text index with tables from the All-tables set (4.2). For comparison, we also experiment with the raw representations using a simple linearization of tables similar to (Oguz et al., 2020). In single data setting for WebQ, we consider combining text with KB graphs from WD-graphs in the single data setting. Different from (Oguz et al., 2020) where a separate entity-linking based retriever is used for KB, we use a single model over the text index with either linearization of raw KB graphs or our verbalized KB graphs. Hence, in our case, both text and data (tables and KB graphs) can be handled by a unified retriever-reader pipeline. In the hybrid data setting for both NQ and WebQ, we use text, All-tables and WD-graphs for retrieval. The statistics of our document index are shown in Table 7 in Appendix A. We create additional retriever training data from NQ-Table-Q and WebQ-KB-Q in a similar fashion as in the text-only setting, so that DPR can better handle additional knowledge. Following (Oguz et al., 2020), we also use the iterative training setup for retriever training. More training details can be found in Appendix B. To evaluate the effectiveness of our UDT-QA for ODQA, we first include recent state-of-the-art ODQA models using text as the only knowledge source, DPR and UnitedQA. We also compare our UDT-QA with recent models using additional structured knowledge, KEALM and UnitK-QA. Following the literature, we report the exact match (EM) score for evaluation. The results are in Table 2. Source Format R20 R100 EM text -80.8 86.1 49.6 + NQ-tables raw 85.2 90.1 51.1 + NQ-tables V 85.5 90.2 51.2 + All-tables raw 85.8 90.7 52.1 + All-tables V 86.0 90.7 52.5 text -78.9 82.3 52.6 + WD-graphs-WebQ raw 83.4 86.1 57.1 + WD-graphs-WebQ V 83.4 85.0 55.7 + WD-graphs raw 82.8 86.1 54.3 + WD-graphs V 82.8 86.7 55.4 Table 3: Impact of document index size over separately trained retriever-reader models (Top for NQ and bottom for WebQ). As we can see, models with additional structured knowledge achieve better performance than text-only models. This indicates that both KB graphs and tables contain complementary knowledge which is either absent in text or harder to be reasoned over. For NQ, although we consider a significantly larger structured knowledge source which is likely to be more challenging, all our models substantially outperform UnitK-QA. As for WebQ, our model achieves competitive performance, although worse than UnitK-QA. We attribute this gap to two possible reasons. First, UnitK-QA uses a separate entity-linking based retriever for KBs which might lead to higher retrieval recall. Second, since WebQ is fully based on FreeBase, using WikiData only in our models likely suffers from mismatch (Pellissier Tanon et al., 2016). Nevertheless, our verbalizer-based models achieve better performances than the corresponding raw format models on both datasets, indicating that the proposed verbalizer is highly effective for tables and KB graphs. In this section, we present analyses over the impact of document index size, the use of additional structured knowledge in a hot-swap setting, comparison to a recent KB-only data-to-text approach in an end-to-end fashion, and manual exam of the verbalized/raw tables for their impact on ODQA. How does the size of document index affect retriever and reader performance? More knowledge is likely to have better coverage of relevant information. On the other hand, larger and noisier index also increases the reasoning complexity. To understand the impact of the increased document index size, we conduct experiments with a restricted setting where only relevant subset of knowledge to the corresponding dataset (a prior) is used for retrieval. Similar to (Oguz et al., 2020), we experiment with the combined document index of text and NQ-tables for NQ. As for WebQ, we keep documents from WD-graphs that contain any of the question entity in WebQ to build WD-graphs-WebQ , and experiment with using text + WD-graphs-WebQ . In addition to EM, we report R20 and R100, evaluating the retrieval accuracy of gold passages in the top-20 and top-100 documents, respectively. The results are reported in Table 3. For NQ, in spite of being more challenging, we see that using All-tables yield substantial improvement in both recall and answer exact match compare to using NQ-tables . This indicates that, with proper training, ODQA models are likely to benefit from enriched knowledge. Although the larger raw form index brings in decent improvement (+1 EM) in terms of reader performance (+ All-tables vs+ NQ-tables ), our verbalized knowledge is more friendly for answer reasoning leading to a more notable QA improvement (+1.3 EM). Different from NQ, we observe that on WebQ the restricted setting with WD-graphs-WebQ achieves better results. We hypothesize that this is likely due to the scale of WebQ dataset. The small amount of WebQ training makes the retriever insufficient to handle large-scale document index. We leave the verification of this hypothesis for future work. Does a text-only retriever-reader model benefit more from verbalized knowledge compare to raw format (hot-swap)? Since both retriever and reader are based on pretrained language models, we hypothesize that they would probably benefit more from the verbalized knowledge due to its sim-Source R20 R100 EM KELM 78.2 85.3 51.5 WD-graphs (Ours) 78.5 85.5 52.0 Table 5: Comparison of verbalized knowledge from our verbalizer and KELM for retriever and reader on WebQ test. This can be particularly useful for a hot-swap setting where both retriever and reader have only seen textual knowledge during training. To verify that verbalized knowledge is more amenable, we carry out a hot-swap experiment here. Specifically, we directly use a DPR model trained on NQ text-only data for additionally indexing both NQ-tables and All-tables . Then, the inference retrieval is performed on the augmented document index for an input question, and a text-only United-QA-E reader trained on NQ is applied for answer inference afterwards. The results are summarized in Table 4. Similar to the previous fully fine-tuned settings, we see that additional knowledge still provide substantial improvements for text-only retriever using either raw or verbalized knowledge. However, the improvement in recall is not reflected in the later reader performance for the raw format, whereas the hot-swap answer inference performance is notably improved with verbalized knowledge. This observation further validates our hypothesis that verbalized knowledge is more beneficial, especially for reader. How does the proposed verbalizer compare to recent data-to-text models? Lastly, we compare our verbalizer with the recently proposed data-to-text generator for converting KB graphs only, KELM (Agarwal et al., 2021). Since both KELM generator and our verbalizer are based on the same partitioned Wikidata, this evaluation can fully re-flect their corresponding generation impacts on ODQA in an end-to-end fashion. Here, we evaluate using our verbalized WD-graphs and the KELM corpus as additional knowledge on WebQ. In particular, we follow the same procedure to train and evaluate our retriever and reader except that we swap the WD-graphs with KELM corpus in data construction and retrieval. Both retriever and reader performances are reported in Table 5. Note that the KELM data-to-text model is customized solely for converting KB graphs and trained with a much larger dataset (about 8M training instances), 1611 Q&A V table Raw table TITLE: List of Star Wars: The Clone Wars episodes Q: star wars .... Nevertheless, consistent with its better extrinsic performance (5), our verbalizer again outperforms the KELM generator in both retrieval and reading, which provides further support for the effectiveness of our approach as a unified interface for ODQA over data and text. What is the impact of verbalized/raw table on ODQA? We manually analyze examples of verbalized and raw tables and the details of annotation can be found in Appendix E. We showcase the examples of verbalized tables and their raw counterpart in Table 6 and discussion their effect on our UDT-QA system. We identify 2 common patterns where raw tables are inferior to verbalized tables, as shown in the first 2 rows of Table 6. In the first example, the concatenated numbers in the raw table can be hard to interpret , and we have to carefully align the row with the header, which is very far away. In the second example, the raw infobox can be in ill-format and very long , making it hard to understand. On the other hand, the verbalized row clearly states the answer evidence by connecting the information in the headers with cell values, making it straightforward to find the answer. At the same time, we also notice the limitation of verbalized tables: table structure loss. We found that raw tables are better at answering ranking questions, as the examples shown in row 3&4 of Table 6. When asked about the top or bottom ranked subject, the model can directly look for evidence from the starting or the end of the table. On the other hand, when the table is verbalized, the model can not rely on such shortcuts because the boundary of rows is not clear and the original structure of the tables are lost . This also suggests a possible direction for future work: to better incorporate the table structure information in verbalization. Data-to-Text Generating text from structured data has been a popular task in NLP. Many dataset have been proposed for this task such as Wikibio (Lebret et al., 2016), Rotowire (Wiseman et al., 1612 2017), WebNLG (Gardent et al., 2017) and E2E (Novikova et al., 2017), where each dataset focuses on a particular domain. More recently, large-scale datasets that contains open-domain examples have been proposed including DART (Nan et al., 2021), TOTTO (Parikh et al., 2020), WikiTableT (Chen et al., 2021a) and GenWiki (Jin et al., 2020). On the modeling side, finetuning the pretrained models typically achieves promising performance (Ribeiro et al., 2020). Wang et al. (2020) propose customized loss functions to reduce model hallucination during generation. Muti-task learning is used to improve model's robustness towards input variations (Hoyle et al., 2021). Chen et al. (2020b) introduce a generalized format and a pretrained model that can generate text from both table rows and knowledge graphs. Most previous work on data-to-text generation have only conducted internal evaluation, using typical generation metrics such as BLEU and ROUGE, hence the data-to-text is considered the target task. In this paper, we argue that different training strategies and evaluation metrics should be adapted when applying data-to-text models to downstream tasks, i.e. ODQA. Related to our work, Agarwal et al. (2021) convert the entire Wikidata to natural language using a finetuned T5 model (Raffel et al., 2020). In this work, we generalize the data-to-text approach for verbalizing both tables and KB graphs in a unified fashion and study the verbalized knowledge on ODQA. QA with Data and Text As the knowledge required to answer the questions may not be available in textual corpus, previous studies have sought to incorporate knowledge from difference sources such as tables and knowledge bases. Min et al. (2019) use Wikidata to expand seed passages found by the retriever and enhance encoded passage representations in the reader. Li et al. (2021) propose a hybrid framework that takes both text and tables as inputs to produce answers and SQL queries. Recently, Chen et al. (2021b) develop the OTT-QA dataset containing questions that require joint reasoning over both tables and text, where the tables and text come from entire Wikipedia. There is also a line of work that studies model architectures for tables specifically or joint encoding of tables and text (Yin et al., 2020; Herzig et al., 2020; Zayats et al., 2021; Glass et al., 2021). However, their focus is not on open-domain QA tasks. Most similar to our work is (Oguz et al., 2020), where they use both tables and Wikidata/Freebase knowledge graph along with Wikipedia text for ODQA. However, they simply linearized structured data without using any verbalizer, thus may suffer from suboptimal input representation. Also, their tables are only mined from original NQ HTMLs, i.e. a constrained setting. In contrast, we consider tables from full Wikipedia which is a much larger set. Additionally, separate retrieval models are used for tables and KB in (Oguz et al., 2020) whereas we develop a unified model over text and data. In this paper, we demonstrated that a unified verbalizer-retriever-reader framework, UDT-QA , for open-domain QA over data and text. We proposed a novel data-to-text paradigm that can largely improve the verbalization effectiveness for downstream knowledge-intensive applications, i.e. open-domain QA, when attaining good intrinsic performances. With the verbalized knowledge, we achieved a new state-of-the-art result for NQ. Remarkably, we showed that simply augmenting the text index with the verbalized knowledge improve the performance without retraining the model. In addition to our method, there are many recently proposed approaches for open-domain QA that are orthogonal. For example, language models specifically optimized for dense retrieval (Gao and Callan, 2021), pretraining on large-scale QA data (Oguz et al., 2021) and hybrid system that consists of retriever, reranker, extractive reader and generative reader (Fajcik et al., 2021). Incorporating those methods may further improve the performance for open-domain QA, and we leave that exploration for future work. Lastly, instead of only considering a sanitized collection of knowledge sources, it is an interesting future direction to scale up the knowledge to web-scale (Nakano et al., 2021; Pik-tus et al., 2021). We would like to thank Ruohong Zhang for helpful discussions and anonymous reviewers for their valuable suggestions on this paper.
abstain abstain abstain method objective result abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain objective abstain abstain abstain abstain objective abstain objective method abstain result method result result abstain abstain method other method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method method method method method abstain method method method method method method method method abstain method abstain method method abstain abstain method method method method method abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other result abstain result result abstain abstain abstain abstain method abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other method method method other other other other other other method other other objective abstain objective objective objective objective objective abstain result abstain other