Numerical reasoning over hybrid data containing both textual and tabular content (e.g., financial reports) has recently attracted much attention in the NLP community. However, existing question answering (QA) benchmarks over hybrid data only include a single flat table in each document and thus lack examples of multistep numerical reasoning across multiple hierarchical tables. To facilitate data analytical progress, we construct a new large-scale benchmark, MULTIHIERTT , with QA pairs over Multi Hier archical T abular and T extual data. MULTIHIERTT is built from a wealth of financial reports and has the following unique characteristics: 1) each document contain multiple tables and longer unstructured texts; 2) most of tables contained are hierarchical; 3) the reasoning process required for each question is more complex and challenging than existing benchmarks; and 4) fine-grained annotations of reasoning processes and supporting facts are provided to reveal complex numerical reasoning. We further introduce a novel QA model termed MT2Net, which first applies facts retrieving to extract relevant supporting facts from both tables and text and then uses a reasoning module to perform symbolic reasoning over retrieved facts. We conduct comprehensive experiments on various baselines. The experimental results show that MULTIHIERTT presents a strong challenge for existing baselines whose results lag far behind the performance of human experts. The dataset and code are publicly available at https://github. com/psunlpgroup/MultiHiertt . In recent years, as key to many NLP tasks such as QA, there is a flurry of works on numerical reasoning over various types of data including textual data (Dua et al., 2019; Amini et al., 2019; Xie and Sun, 2019) and tabular data (Moosavi et al., 2021; Suadaa et al., 2021). More recently, numerical Figure 1: An example of MULTIHIERTT : The system needs to first locate which segment got the most funds in 2017 in the second hierarchical table, then select relevant numbers from the first hierarchical table and generate the correct reasoning program to get the answer. reasoning over hybrid data containing both textual and tabular content (Zhu et al., 2021; Chen et al., 2021) has attracted much attention. For example, 6588 the FinQA dataset (Chen et al., 2021) focuses on questions that require numerical reasoning over financial report pages, e.g., "What portion of the total identifiable net assets is in cash?". Such questions need the system to locate relevant cells in the tabular content and then perform a division operation to get the final answer. However, existing QA datasets over hybrid data only contain a single flat table in each document (Zhu et al., 2021; Chen et al., 2021). Therefore, they lack examples that require multi-step reasoning processes across multiple paragraphs and hierarchical tables. Hierarchical tables are widely used in scientific or business documents. A hierarchical table usually contains multi-level headers, which makes cell selection much more challenging because it requires multi-level and bi-dimensional indexing techniques. For instance, consider the example of our proposed dataset MULTIHIERTT in Figure 1, each table contains both column headers and row headers, which are hierarchical in nature. And ignoring the row / column headers or not reasoning on the entire header hierarchy may lead to the wrong result. For instance, in the given example, if the system simply searched for cells with a flat row header containing "Product" and "Service" and column header containing "2018", it may mistakenly return the value 2,894 and 382 appearing in the beginning of the first table. Additionally, in real life, when analyzing financial reports, professionals such as analysts or investors often refer to multiple hierarchical tables and multiple paragraphs to obtain conclusions. For instance, finding "the segments with most funds in 2017" requires the system to locate and perform numerical reasoning on the second hierarchical table. Then the system should use the results gained from the second table to reason on the first table. However, existing QA datasets lack such examples of reasoning across multiple tables. To address these shortcomings, we present MULTIHIERTT : an expert-annotated dataset that contains 10,440 QA pairs, along with annotations of reasoning processes and supporting facts. To the best of our knowledge, MULTIHIERTT is the first dataset for solving complicated QA tasks over documents containing multiple hierarchical tables and paragraphs. In addition, to address the challenge of MULTIHIERTT , we propose MT2Net to first retrieve supporting facts from financial reports then generate executable reasoning programs to answer the questions. Our experiments show that MT2Net outperforms all other baselines and achieves 38.43% F1 score. However, all models still lag far behind the performance of human experts with 87.03% in F1. It demonstrates MULTIHIERTT presents a strong challenge for existing baseline models and is a valuable benchmark for future research. The main contribution of this work can be summarized as follows:  We propose a new large-scale dataset MULTIHIERTT . It contains 10,440 examples along with fully annotated numerical reasoning processes and supporting facts. A strict quality control procedure is applied to ensure the meaningfulness, diversity, and correctness of each annotated QA example. Compared with existing datasets, each document in MULTIHIERTT contains multiple hierarchical tables and longer unstructured text. A more complex reasoning process across multiple tables and paragraphs is required to correctly answer the question. We propose a novel QA model, MT2Net. The model first applies facts retrieving to extract relevant supporting facts from both hierarchical tables and text. And it then uses a reasoning module to reason over retrieved facts. Comprehensive experiments are conducted on various baselines. The experimental results demonstrate that the current QA models still lag far behind the human expert performance, and further research is needed. Question Answering Benchmark There are numerous QA datasets focusing on text, ta-ble/knowledge base (KB), and hybrid data. SQuAD (Rajpurkar et al., 2016) and CNN/Daily Mail (Hermann et al., 2015) are classic datasets for textual data. Table/KB QA datasets mainly focus on structured tables (Pasupat and Liang, 2015; Zhong et al., 2017; Yu et al., 2018; Nan et al., 2022) and knowledge bases (Berant et al., 2013; Yih et al., 2015; Talmor and Berant, 2018; Xie et al., 2022). And some recent works focus on reasoning over more complex tables including hierarchical tables (Cheng et al., 2021b; Katsis et al., 6589 QA Dataset Textual & Tabular Data / Doc (DB) NumericalReasoning # Doc (DB) # Questions Avg. 2021). More recently, there are also some pioneering studies working on QA over hybrid data. Specifically, HybridQA (Chen et al., 2020), TAT-QA (Zhu et al., 2021), and FinQA (Chen et al., 2021) focus on both textual and tabular data, while MMQA (Talmor et al., 2021) focus on QA over text, tables, and images. In addition, reasoning including numerical reasoning and multi-hop reasoning has gained attention lately. For example, DROP (Dua et al., 2019) is a machine reading comprehension benchmark that requires numerical reasoning on text data. HotpotQA (Yang et al., 2018) and HybridQA (Chen et al., 2020) are datasets requiring multi-hop reasoning. Numerical Reasoning Numerical reasoning plays an important role in different NLP tasks (Dua et al., 2019; Zhang et al., 2021; Chen et al., 2021; Zhu et al., 2021). To enhance the model's numerical reasoning ability, some work adapt standard extractive QA models with specialized modules to perform numerical reasoning (Ran et al., 2019; Hu et al., 2019). Recent work also focus on probing and injecting numerical reasoning skills to pretrained language models (Geva et al., 2020; Lin et al., 2020; Zhang et al., 2020; Berg-Kirkpatrick and Spokoyny, 2020). Meanwhile, various benchmarks and models are proposed to solve math word problems (Koncel-Kedziorski et al., 2016; Xie and Sun, 2019; Amini et al., 2019; Hendrycks et al., 2021; Hong et al., 2021; Cobbe et al., 2021). The most recent numerical reasoning QA benchmark over hybrid data are FinQA (Chen et al., 2021) and TAT-QA (Zhu et al., 2021). Financial NLP Financial NLP has attracted much attention recently. There have been various application in different tasks like risk management (Han et al., 2018; Theil et al., 2018; Nourbakhsh and Bang, 2019; Mai et al., 2019; Wang et al., 2019), asset management (Filgueiras et al., 2019; Blumenthal and Graf, 2019), market sentiment analysis (Daudert et al., 2018; Tabari et al., 2018; Buechel et al., 2019), financial event extraction (Ein-Dor et al., 2019; Zhai and Zhang, 2019) and financial question answering (Lai et al., 2018; Maia et al., 2018). More recently, pre-trained language models are presented for finance text mining (Araci, 2019; Yang et al., 2020). The most relevant work to us is FinQA (Chen et al., 2021) and TAT-QA (Zhu et al., 2021), which both construct a QA dataset acquiring numerical reasoning skills on financial reports with tabular data. MULTIHIERTT are deployed based on the FinTabNet dataset (Zheng et al., 2021), which contains 89,646 pages with table annotations extracted from the annual reports of S&P 500 companies. For each table contained, the FinTabNet dataset provides a detailed HTML format annotation, in which table hierarchies and cell information such as text and 6590 What is the total amount of options granted and accepted in 2007 for exercise price? The raw data is filtered as follows: First, we extract documents with 1 to 4 pages and 2 to 6 tables from FinTabNet. Second, we filter out the documents with limited textual contents. Third, as we aim for the numerical reasoning ability, we also exclude documents with tables containing little numerical information. Then, we use a pre-processing script to extract the hierarchical structure of each HTML-format table. And we ignore those tables that cannot be handled by the pre-processing script. As a result, a total of 4,791 documents were selected for further annotation. For each document selected in 3.1, the annotators are required to compose one or two QA examples along with detailed annotation. The process of annotating each QA example is as follows: 1) The annotators are first asked to compose a complex question that requires numerical reasoning and is meaningful for helping novices understand the annual reports. The annotators are encouraged to compose questions that require the information from both the textual and tabular content or from multiple tables. 2) For those questions requiring numerical expression, the annotators are then asked to write down the reasoning programs to answer the question. In detail, the annotators are asked to elaborate on the operation steps to answer the question. The definitions of all operations are shown in Table 7 in Appendix. 3) They are also required to mark all the supporting facts from tabular and textual content for each question. Strict quality control procedures are designed to ensure the quality of dataset annotation, especially the diversity and meaningfulness of proposed questions. The human evaluation scores and inter-evaluator agreements are reported in Table 2. Expert Annotators To help improve the annotation process, we first enroll five experts with professional experience in finance. During annotation, they are asked to provide feedback regarding the task instructions and the user experience of the annotation interface, based on which we iteratively modify the annotation guideline and interface design. In the stage of crowd-sourced annotation, we hire 23 graduate students (14 females and 9 males) majoring in finance or similar discipline. Before starting the official annotation process, each annotator is given a two-hour training session to learn 6591 the requirements and the annotation interface. Annotation De-Biasing As suggested in previous research (Kaushik and Lipton, 2018; Clark et al., 2019; Jiang and Bansal, 2019; Yang et al., 2022), consider annotation bias of QA benchmarks is of great significance. During the pilot annotation period, we found that when generating question-answering pairs, annotators may prefer simpler ones. To solve this issue, we use thresholds to restrict the proportions of questions with different numbers of numerical reasoning steps. Meanwhile, the proportions of questions with span selection answer types are set to  20%. To further increase the diversity of question-answer pair annotation, we also select and include 2,119 QA examples from FinQA (Chen et al., 2021). Multi-Round Validation To further ensure the diversity and correctness of proposed question-reasoning pairs, each document is assigned to three annotators and one verifier in order. For annotators, each is required to first validate the previous annotator's annotation and fix the mistakes if there are. Then, they are asked to create one or two more question-reasoning pairs that are different from the existing ones. After each annotator finishes tasks, we assign another verifier with good performance on this project to validate all the annotations. Core statistics of MULTIHIERTT are reported in Table 3. Table 1 shows a comprehensive comparison of related datasets. MULTIHIERTT is the first dataset to study numerical reasoning questions over hybrid data containing multiple hierarchical tables. Compared with TAT-QA and FinQA, documents in MULTIHIERTT contain longer unstructured input text and multiple tables, making the evidence retrieving and reasoning more challenging. And MULTIHIERTT has diverse and complex questions, as illustrated in Figure 2. We also analyze supporting facts coverage for each question. In MULTIHIERTT , 1) 10.24% of the questions only require the information in the paragraphs to answer; 2) 33.09% of the questions only require the information in one table to answer; 3) 7.93% require the information in more than one table but without paragraphs to answer; 4) 48.74% require both the text and table information to answer, and among them, 23.20% required the information in more than one table. The average number of annotated supporting facts are 7.02. Meanwhile, among those questions with annotated numerical reasoning programs, 28.94% of them have 1 step; 37.76% of them have 2 steps; 15.21% of them have 3 steps; and 18.10% of them have more than 3 steps. As a result, the average number of numerical reasoning steps is 2.47. To address the challenge of MULTIHIERTT , we propose a framework named MT2Net. Figure 3 gives an overview of our proposed model. MT2Net first applies fact retrieving module to extract relevant supporting facts from the hierarchical tables and paragraphs. Then, a reasoning module is adapted to perform reasoning over retrieved facts and get the final answer. Fact Retrieving Module The whole input text in each document of MULTIHIERTT can exceed 3,000 tokens and contain many numbers, which is beyond the capability of the current popular QA models (Devlin et al., 2019; Liu et al., 2019). Therefore, we employ a fact retrieving module to first retrieve the supporting facts from the documents. Previous works on hybrid datasets (Zhu et al., 2021; Chen et al., 2021; Li et al., 2021) use templates to flatten each row of the table into sentences. And our facts retrieving module applies similar ideas. However, different from other hybrid datasets, most tables in MULTIHIERTT are hierarchical. Therefore, we turn each cell into a sentence, along with its hierarchical row and column headers. For example, the first data cell in the first table in Figure 1 is translated as "For Innovation Systems of Segment, sales of product in 2018, Year Ended December 31 is 2,894". bi-classifier (Devlin et al., 2019). During the inference stage, the topn sentences are retrieved as supporting facts. They are reordered according to the order of appearance in the original document. Then they will serve as input to reasoning module. Reasoning Module We first use pre-trained LMs to encode the retrieved sentences from the facts retrieving module. Then, we divide the answers into two types: arithmetic program and span. For each answer type, we use a unique sub-module to calculate the conditional answer probability P ( answer | type ) : Program sub-module : The structure is similar with the program generator of FinQANet (Chen et al., 2021). The sub-module aims to generate the executable program to answer the question. Specifically, an LSTM is used for decoding. At each decoding step T , the LSTM can generate one token from 1) the numbers from the retrieved, 2) pre-defined operators, and 3) the tokens already generated in the previous steps. After the completion of generation, the sub-module will execute the generated programs and get the predicted answer. Span sub-module : The span sub-module aims to select the predicted span candidate, which is a span of retrieved sentences. The answer probability is defined as the product of the probabilities of the start and end positions in the retrieved evidence. Meanwhile, an extra output layer is used to predict the probability P ( type ) of each answer type. In particular, we take the output vector [CLS] from LMs as input to compute the probability. In the training stage, the final answer probability is defined as the joint probability over all feasible answer types, i.e., (cid:80) type P ( type )  P ( answer | type ) . Here, both P ( type ) and P ( answer | type ) is learned by the model. In the inference stage, the model first selects the most probable answer type and then uses corresponding sub-modules to predict the answer. TAGOPTAGOP 1 is the baseline model for TAT-QA dataset (Zhu et al., 2021). It first uses sequence tagging with the InsideOutside tagging (IO) approach to extract supporting facts. Then an operator classifier is applied to decide which operator is used to infer the final answer via extracted facts. Different from ours, TAGOP can only perform symbolic reasoning with a single type of pre-defined aggregation operators (e.g. change Ratio, division), and might fail to answer complex questions requiring multi-step reasoning. FinQANet FinQANet 2 is the baseline model for FinQA dataset (Chen et al., 2021). It first uses a BERT-based retriever to take the topn supporting facts. Then a program generator is applied to generate the reasoning programs to get the final answers. Different from ours, FinQANet ignores the hierarchical structure of tables when linearizing each row of a table. And it is not designed to answer span selection questions. Longformer + Reasoning module To demonstrate the necessity of breaking up models into facts retrieving and reasoning modules, we directly use the pre-trained Longformer-base 3 (Beltagy et al., 2020) as the input encoder in the reasoning module, and encode the whole document. Fact Retrieving Module + TAPAS We employ TAPAS (MASKLM-base) 4 (Herzig et al., 2020; Eisenschlos et al., 2020) as a baseline over tabular data. TaPas is pretrained over large-scale tables and associated text from Wikipedia jointly. To finetune it, we use the table with most supporting facts along with the answer as input for each example. For the inference stage, the table with most portion of top-15 retrieved facts is used as input. Fact Retrieving + NumNet NumNet+ 5 (Ran et al., 2019) has demonstrated its effectiveness on the DROP dataset (Dua et al., 2019). It designs a NumGNN between the encoding and prediction module to perform numerical comparison and numerical reasoning. However, NumNet+ only supports addition and subtraction when performing symbolic reasoning, thus cannot handle those complex questions requiring operators such as division. Fact Retrieving Module + Seq2Prog A Seq2Prog architecture adopted from baseline of MathQA dataset (Amini et al., 2019) is used as the reasoning module. Specifically, we use a biLSTM encoder and an LSTM decoder with attention. For the fact retrieving module, we use BERT-base as the classifier. Since most of the examples in our dataset have less than 7 supporting facts (89.3%), and we find that longer inputs might lower the performance of the reasoning module, we take the top10 retrieving facts as the retriever results. For the reasoning module, we experiment on using BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as the encoder. We use the Adam optimizer (Kingma and Ba, 2014) for all models. The 3 https://github.com/allenai/longformer 4 https://github.com/google-research/ tapas 5 https://github.com/llamazing/numnet_ plus Dev Test EM F 1 EM F 1 Longformer + Reasoning 2.71 6.93 2.86 6.23 Facts Retrieving + TAPAS 8.94 10.70 7.67 10.04 Facts Retrieving + NumNet 10.32 12.59 10.77 12.02 TAGOP (RoBERTa-large) 19.16 21.08 17.81 19.35 Facts Retrieving + Seq2Prog 26.19 28.74 24.58 26.30 FinQANet (RoBERTa-large) 32.41 35.37 31.72 33.60 MT2Net (BERT-base) 33.68 35.94 32.07 33.67 MT2Net (BERT-large) 34.03 36.13 33.25 34.98 MT2Net (RoBERTa-base) 35.69 37.81 34.32 36.17 MT2Net (RoBERTa-large) 37.05 39.96 36.22 38.43 Human Expert Performance   83.12 87.03 Table 4: Performance of MT2Net compared with different baseline models on the dev and test sets of MULTIHIERTT . training of all models is conducted on RTX 3090s. All the implementation of LMs is based on the hug-gingface transformers library. To ensure fairness, we set batch size as 32 for all baseline models. For Evaluation Metrics, following TAT-QA (Zhu et al., 2021), we report exact matching (EM) and adopted numeracy-focused F 1 (Dua et al., 2019). To test the performance of the human expert on MULTIHIERTT , we invite another two professionals. We randomly sampled 60 examples from the test set, and ask them to answer the questions individually within three hours. The results are reported in the last row of Table 4. Table 4 summarizes our evaluation results of different models. We use the same fact retrieving results for all "Retrieving + Reasoning" models. For the fact retrieving module, we have 76.4% recall for the top10 retrieved facts and 80.8% recall for the top15 retrieved facts. Necessity of applying retrieving-reasoning pipeline Directly using an end-to-end pretrained Longformer model to replace a retrieving module falls far behind. This makes sense because longer input contains much irrelevant numerical information, which makes the reasoning module difficult to learn. worse than MT2Net because they ignore the table's hierarchical structure in the retrieving part. Different from ours, which flatten each cell with its header hierarchical structures, both TAGOP and FinQANet flatten each table by rows, losing the table's hierarchical structure information. Necessity of an effective reasoning module Most questions in MULTIHIERTT require models to perform multi-step reasoning and integrate different kinds of operators. Generally, the reasoning module generating reasoning programs to get answers performs better than directly generating answers by end-to-end method, i.e. adopted TAPAS. Both adopted NumNet and TAGOP perform much worse than MT2Net because they only support limited symbolic reasoning. Specifically, TAGOP can only perform with a single type of pre-defined aggregation operator for each question, and NumNet only supports addition and subtraction operators when performing symbolic reasoning. By contrast, MT2Net performs better than FinQANet and Seq2Prog because it applies different sub-modules to answer questions with different answer types. The results also show that larger pre-trained models have better performance. This is because they are pre-trained on more financial corpus. However, all the models perform significantly worse than human experts, indicating MULTIHIERTT is challenging to state-of-the-art QA models and there is a large room for improvements for future research. To guide the future directions of model improvement, various performance breakdown experiments on the test set are conducted using the MT2Net (RoBERTa-large) model. Table 5 shows the results. Generally, the model has a much lower accuracy on questions with more than two numerical reasoning steps. Meanwhile, the model performs poorly on questions requiring cross-table supporting facts. We further investigate the proposed MT2Net by analyzing error cases. We randomly sample 100 error cases from the results of the MT2Net (RoBERTa-large) model on the test set, and classify them into four main categories as shown in Table 6, along with examples. The analysis shows that around 64% error (Wrong Operand/Span+Missing Operand) is caused by the failure to integrate the supporting facts correctly. Meanwhile, the current model fails to integrate external finance knowledge to answer questions. Although the proposed MT2Net model outperforms other baseline models, it still performs significantly worse than human experts, which reflects the challenge of MULTIHIERTT . Primarily, we find that models do not perform well on certain types of questions: 1) questions requiring reasoning across multiple tables; 2) questions requiring multi-step reasoning; 3) questions requiring reasoning over tables with complex hierarchical structures; and 4) questions requiring external financial knowledge. four main directions of work may be workable: 1) designing a specialized module to handle multi-table reasoning; 2) decomposing a complex question requiring multi-step reasoning into several simpler sub-questions that QA models can handle (Perez et al., 2020; Chen et al., 2020); 3) applying a more advanced table-encoding method. For example, a pre-trained model with specialized table structure-aware mechanisms (Wang et al., 2021; Cheng et al., 2021a; Yang et al., 2022) can be utilized in the facts retrieving module to better understand hierarchical tables; and 4) leveraging structured knowledge (Xie et al., 2022) to inject external financial knowledge to models. We have proposed MULTIHIERTT , a new large-scale QA dataset that aims to solve complicated QA tasks that require numerical reasoning over documents containing multiple hierarchical tables and paragraphs. To address the challenge of MULTIHIERTT , we introduce a baseline framework named MT2Net. The framework first retrieves supporting facts from financial reports and then generates executable reasoning programs to answer the question. The results of comprehensive experiments showed that current QA models (best F 1 : 38.43 % ) still lag far behind the human expert performance (F 1 : 87.03 % ). This motivates further research on developing QA models for such complex hybrid data with multiple hierarchical tables. Data in MULTIHIERTT is collected from the FinQA dataset (Chen et al., 2021) and FinTabNet dataset (Zheng et al., 2021). FinQA is publicly available under the MIT license 6 . FinTabNet is publicly available under the license CDLA-Permissive-1.0 7 . Both licenses permits us to compose, modify, publish, and distribute additional annotations upon the original dataset. For the internal annotation of MULTIHIERTT , each expert is paid $20 per hour. For the external annotation, we hire 23 graduate students majoring in finance or similar disciplines. We regard creating one question-reasoning pair, or validating one document's annotation as a unit task. And we pay around $1.1 for each unit task. Averagely, an annotator can finish 7 unit tasks per hour after training 6 https://opensource.org/licenses/MIT 7 https://cdla.dev/permissive-1-0/ and practicing. And the hourly rates are in the range of $6 and $9 based on the different working speed (above the local average wage of similar jobs). In total, the approximate working hours to annotate MULTIHIERTT dataset is 1500 hours. The whole annotation work lasts about 70 days. We appreciate all the annotators' efforts to construct MULTIHIERTT . And we would like to thank the anonymous reviewers and action editors for their constructive discussions and feedback.
abstain abstain objective abstain abstain abstain abstain abstain objective method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain method objective objective result abstain abstain objective objective objective objective objective objective objective objective objective objective other other other other other other other other other other other other other other other other other other method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain other abstain abstain abstain other other