Tables store rich numerical data, but numerical reasoning over tables is still a challenge. In this paper, we find that the spreadsheet formula, a commonly used language to perform computations on numerical values in spreadsheets, is valuable supervision for numerical reasoning in tables. Considering large amounts of spreadsheets available on the web, we propose FORTAP , the first exploration to leverage spreadsheet formulas for table pretraining. Two novel self-supervised pretraining objectives are derived from formulas, numerical reference prediction (NRP) and numerical calculation prediction (NCP). While our proposed objectives are generic for encoders, to better capture spreadsheet table layouts and structures, we build FORTAP upon TUTA, the first transformer-based method for spread-sheet&web table pretraining with tree attention. FORTAP outperforms state-of-the-art methods by large margins on three representative datasets of formula prediction, question answering, and cell type classification, showing the great potential of leveraging formulas for table pretraining. The code will be released at https://github.com/microsoft/TUTA_ table_understanding . Tables store rich numerical data, so a wide range of tasks require numerical reasoning over (semi-)structured tabular context, such as question answering over tables (Chen et al. , 2021b; Zhu et al. , 2021; Cheng et al. , 2021), table-to-text (Suadaa et al. , 2021; Moosavi et al. , 2021; Cheng et al. , 2021), spreadsheet formula prediction (Chen et al. , 2021a), and table structure understanding (Koci et al. , 2019). Take Table#2 in Figure 1 as an example, both suggesting the formula (C4-B4)/B4 for cell D4 and answering 0.61% to the question require  The first two authors contribute equally. numerical reasoning capabilities of (1) understanding the contextual meaning of individual numerical cells, e.g., 11.49 at B4 and 11.56 at C4 are populations of Belgium in 2019 and 2020; (2) inferring calculational relationships of numerical cells, e.g., percentage change from 11.49 to 11.56. As Figure 1 shows, same capabilities also benefit table structure recognition and table-to-text. So it's a fundamental need to empower table modeling with stronger numerical reasoning capabilities. However, it is challenging to endow a tabular model with robust numerical reasoning capabilities. First, understanding a local numerical cell needs dimension inference (Chambers and Erwig, 2008), unit inference (Shbita et al. , 2019), and index inference (Dong et al. , 2019a), e.g., popula-tion (dimension), million (unit), 2020 (index), and Belgium (index) jointly describe 11.56 in Figure 1. It is non-trivial concerning the great flexibility of table semantic structures (Wang et al. , 1150 2021b). Second, calculational relationships among two or more numerical cells are various and often compositional, e.g., F1 Score = 2  (Recall  Precision) / (Recall + Precision) in machine learning papers and Profit Margin = Net Income / Sales in financial reports. To make matters more challenging, human labeling for numerical reasoning in relevant tasks (Chen et al. , 2020; Suadaa et al. , 2021; Koci et al. , 2019) is labor-intensive and error-prone, largely restricting the generalization ability of large models that are rather data-hungry. Recently, table pretraining on large amount of unlabeled tables shows promising results on table understanding and reasoning. Self-supervised objectives are derived from tables and text such as Masked Language Models (MLM) (Herzig et al. , 2020), masked column prediction (Yin et al. , 2020), masked entity recovery (Deng et al. , 2020b), cell cloze and corrupt detection (Wang et al. , 2021b; Tang et al. , 2020; Iida et al. , 2021), table-text matching and alignment (Wang et al. , 2021a,b; Deng et al. , 2020a). However, numerical and calculational relationships of cells lack sufficient attention. Then (Yoran et al. , 2021) and (Liu et al. , 2021; Yu et al. , 2020) synthesize questions and SQL queries, respectively, as training corpus for reasoning purpose, but SQL is only applicable to database-like relational tables, and importantly, it's challenging to ensure synthesized questions and SQLs be realistic, meaningful, and diverse. Gladly, tens of millions of real spreadsheet formulas are publicly available on the web and can be valuable for numerical reasoning in tables. The spreadsheet formula is an expressive yet simple language consisting of operators (e.g., +,/,% ), functions (e.g., SUM,MAX,COUNT ), referenced cells (e.g., B4 ), and constant values (e.g., 100 ) (Aival-oglou et al. , 2015). Since writing the formula does not require formal programming education, it's widely used by non-programmers such as business professionals or other kinds of domain specialists whose jobs involve computational tasks. So spreadsheet formulas cover real numerical calculations in a great variety of domains. To this end, we propose FORmula-driven TAble Pretraining (FORTAP ) for numerical reasoning. One should master two basic concepts to use the formula language: cells as variables and opera-tors/functions as relationships between variables. So we explicitly decompose information in formulas into numerical reference and numerical calculation and devise two complementary tasks. Given a table as well as a formula cell in it, we mask the formula and then (1) the model classifies whether header A references header B  (we consider that header A references header B  if the formula cell belonging to header A references a numerical cell belonging to header B , as illustrated in Figure 2); (2) the model predicts the operator/function of two or more referenced numerical cells. Furthermore, to better encode and represent formulas, we also apply MLM to the token sequence of formulas. Considering the flexibility of table structures in spreadsheets, we base FORTAP on TUTA (Wang et al. , 2021b), the first transformer-based method for spreadsheet tables with carefully-designed textual, numerical, positional, and formatting embedding layers. Importantly, its tree-based position encoding and attention are highly effective in representing generally structured tables. TUTA is pretrained with MLM, cell cloze, and table-text matching. Experiment results on three tasks demonstrate that the significance of leveraging formulas for table pretraining. For formula prediction, FORTAP achieves 55 . 8% top-1 accuracy, significantly surpassing TUTA ( 48 . 5% ), TaPEx ( 43 . 2% ), and SpreadsheetCoder ( 40 . 4% ) on Enron. For table question answering, TUTA achieves comparable accuracy with the best system on HiTab. After pretraining with formulas, FORTAP delivers a huge improvement of +6 . 3% as over previous SOTA, comparable to TaPEx. For cell type classification, on dataset DeEx, FORTAP largely improves TUTA by +6 . 6% on derived type and +3 . 2% on overall Macro-F1. TUTA (Wang et al. , 2021b) is the first pretraining architecture for spreadsheet tables. It is effective in capturing table semantic structures, achieving SOTA results on cell type and table type classification. As mentioned in Section 1, understanding table semantic structures is critical to numerical reasoning, so we choose TUTA to be the encoder of FORTAP . Since our pretraining tasks are generic for encoders of tables, future works can also explore other encoders such as (Herzig et al. , 2020). Header Recognition. Headers usually provide short yet informative descriptions of table contents in Natural Language (NL), so TUTA leverages the detected header regions and hierarchies, as pre-1151 sented in Section 2.2. (Chen et al. , 2021a) also shows that using headers (even without considering hierarchies) greatly helps formula prediction. FORTAP follows to place detected headers in inputs. Architecture. TUTA bases on BERT (Devlin et al. , 2019) with several enhancements: (1) a positional encoding layer based on a unified bi-dimensional coordinate tree to describe both the spatial and hierarchical information of cells; (2) a number encoding layer to encode magnitude, precision, the first digit, and the last digit; (3) a tree-based attention mechanism that enables local cells to aggregate their structurally neighbouring contexts within a tree-based distance threshold. Model Input/Output. The input consists of a table T and optional NL texts C . By traversing the cell matrix of a table from left to right and from top to bottom, the input is linearized to  [CLS] , C 0 , ..., CK  1 , [SEP] , T (0 , 0) , [SEP] , T (0 , 1) , ..., [SEP] , T ( M  1 ,N  1) , where K is the token length of NL texts, and M and N are the numbers of rows and columns of the table, respectively. Note that T ( i,j ) refers to the token sequence of the cell string in the ( i + 1) th row and ( j + 1) th column, and each token has token, number, position, and format input embeddings. The output of the encoder contains token-level, cell-level, and table-level embeddings. FORTAP follows these input/output settings except when inputting formula token sequence. Spreadsheet Source and Preprocessing. We use the same spreadsheet table corpus as TUTA: (1) 13 . 5 million public spreadsheet files are crawled from 1 . 75 million websites; (2) table ranges and headers are detected using TableSense (Dong et al. , 2019b,a); (3) header hierarchies are extracted with effective heuristics; (4) extreme size tables are filtered out; (5) duplicated tables are discarded. In the end, 4 . 5 million spreadsheet tables are left. Formula Preprocessing. Spreadsheet Formula is a widely-used end-user language for table organization and calculation. A formula consists of four types of formula tokens: operator (e.g., +,/,% ), functions (e.g., SUM ), referenced cells (e.g., B4 ) and constant values (e.g., 100), which we denote as OP , FUNC , CELL and CONST in the rest part of the paper. We use XLParser (Aivaloglou et al. , 2015), a highly-compatible formula parser with compact grammar, to analyze formula. In this way, we derive the AST of each formula (an example AST in Figure 2) and the type of each formula token. Since we focus on single table setting, we discard the cross-table, cross-sheet, and cross-file formulas. Formulas with Array or User-Defined-Function are also discarded. The absolute reference sign $ is deleted from formula strings, without changing their meanings. We only keep the first five occurrences of formulas in the same row/column because some spreadsheets contain hundreds of duplicated or dragged formulas in one row/column, which are inefficient for training. Formulas are linearized as formula token sequences in prefix representation of AST following SpreadsheetCoder (Chen et al. , 2021a). Finally, 10.8 million formulas are derived. As mentioned in Section 1, empowering table modeling with stronger numerical reasoning capabilities is a fundamental need. Spreadsheet formulas naturally contain information of numerical references ( CELL ) and calculations ( OP / FUNC ), motivating us to devise effective tasks to leverage them for numerical-reasoning-aware pretraining. Based on information parsed from the formula expression, we carefully devise two complementary objectives, Numerical Reference Prediction (NRP) and Numerical Calculation Prediction (NCP), to exploit the reasoning process behind referencing local cells (as operands) and applying calculations (on operands), respectively. Meanwhile, to get better representations of the spreadsheet formula, which could be further used in downstream applications like formula error detection (Cheung et al. , 2016), we extend MLM (De-vlin et al. , 2019) from NL contexts to formulas. Figure 2 gives an illustration of these tasks. Numerical Reference Predication (NRP) We consider header A references header B  in a table if: in a formula, the formula cell (cell with formula) belonging to header A references a cell belonging to header B . Take the table in Figure 2 as an example, the header %Increase references headers 2016 and 2021 since E3 in column %Increase references C3 and D3 in columns 2016 and 2021 . We let the model learn header reference relationship since a cell belonging to a referenced header is more likely to be involved in the calculation. It is important but usually unknown as a priori, especially when tables are from diverse or unfamiliar domains. Note that we use header cells instead of data cells in this task since headers provide high-1152 (D3-C3)/C3 Numerical Reference Prediction Numerical Calculation Prediction Formula MLM ( %Increase , 2016 ) ( %Increase , Vegetable ) D3 C3 / C3 Predict Calculation (D3 C3) / (D3-C3)/C3 ( %Increase , 2021 ) +  Example table Formula-based Prompt: Symphony %Increase passages cheer Onion over. level descriptions of the data (Chen et al. , 2021a) and thus header reference relationships have more generic semantics across tables. Given extracted header regions and hierarchies in corpus preprocessing, we first formulate NRP as a binary classification task over header pairs: given a formula cell t f and its referenced cells { t ( i ) p } , we first find their non-shared headers h f (for t f ) and { h ( i ) p } (for { t ( i ) p } ), then we group them as positive pairs { ( h f , h ( i ) p ) } . Usually a formula cell shares a header with referenced cells in the same row/column (e.g., in Figure 2, Onion is the shared header for E3 , C3 , D3 ). As it does not reflect header reference relationships, we exclude the shared header in this task. The negative pairs { ( h f , h ( i ) n ) } are sampled among those unreferenced headers on the same direction (either on top or left headers) of h f . Number of negative samples is at most 3:1 to positive ones to balance samples. The binary classification probability of the i th pair p ( i ) = f ( h f , h ( i ) p/n ) , where h is the header cell embedding derived by the encoder and f (  ) is a two-layer binary classification module. To inject table-text joint reasoning skills into FORTAP , which TUTA does not excel at, we further extend NRP task to table-text setting. Given a table with a formula cell, we first construct a formula-based prompt as context by picking 1 to 10 tokens randomly from the vocabulary as a noisy sentence and then inserting the row and column header of formula cell into it at random positions. Next, we jointly input the formula-based prompt and the table, and the task is to classify (1) formula header cell, (2) formula cell, (3) reference header cell, (4) other cells from the table. To precisely classify these cells, model needs to first align formula header cells in table with prompt (alignment skill), then infer the intersection cell of formula header cells as formula cell (spatial reasoning). Finally, it has to identify referenced cells (numerical reasoning) by the formula headers. The NRP loss L nr is calculated as the sum of binary cross entropy loss and multi-class cross entropy loss under table-only and table-text setting. Numerical Calculation Prediction (NCP) Given data cells as operands, a model then needs to find out which operators/functions should be applied. For example, in Figure 2, subtraction and division are applied on C3 and D3 in the formula. We hope the model can speculate the target op-erator/function based on the semantics, numeracy, and positions of given operands (data cells). Thus, we design the task to predict the operator/function for a group of data cells with their contextual cell embeddings produced by the encoder. We formulate it as a multi-class classification task: given a formula and its AST parsed in pre-rpocessing, we select the operators/functions { o ( i ) } satisfying that all direct children nodes { d ( j ) } ( i ) on the formula AST of o ( i ) are in CELL type with integer or float data. The probability of predicting the operator/function of these data cells is p ( i ) = f ( POOL ( { d ( j ) } ( i ) )) , where d is the output cell embedding by the encoder, f (  ) is a two-layer classification module, and POOL is a mean-pooling layer. Note that we only include the op-erator/function o whose all direct children nodes are in CELL type in this task, because otherwise some descendant data cells will first be calculated via other operators/functions and thus have indirect connections with o (e.g., in Figure 2,  /  is 1153 not a target operator since its left child is an operator   ). We include 17 common calculation operators/functions (see Appendix A) covered in spreadsheet formulas in this task. The NCP objective L nc is the multi-class cross entropy loss. Formula MLM To encode formulas, we expand 41 tokens in the vocabulary for all four formula token types, covering 99 . 1% formulas in corpus. Added tokens are listed in Appendix A. Note that a special case is the CELL type, like D4 , because it references another cell. Since referenced cells can be anywhere in a large table, it is infeasible to explicitly insert all cell positions into the vocabulary. Thus, for CELL type token in formula, we use a [RANGE] tag as input token and copy all cell-level embeddings (position, format, numeric, ...) from the referenced cell to this CELL type token. We then apply MLM to formula tokens. Masking and recovering operators/functions is straightforward. When masking or recovering a referenced cell in a formula, we need to avoid label leakage from embeddings of the referenced cell. Thus, to mask a referenced cell, besides using the [MASK] token embedding, the number embedding is set to default to mask the number, and the position and format embeddings are set to the same as the formula cell. To recover a masked referenced cell t r , the cell t ( i ) in input sequence with the highest probability p ( i ) = Softmax ( f ( t r , t ( i ) )) is selected as the predicted cell, where t is output cell embedding of the encoder and f (  ) is a two-layer classification module. The objective L fmlm is calculated as the sum of cross entropy loss over operator/function recovery and referenced cell recovery. Finally, the total pretraining objective is L = L nr + L nc + L fmlm (1) 4 Experiments In this section, we describe the pretraining details and validate the effectiveness of FORTAP on three downstream tasks: formula prediction, question answering, and cell type classification. The statistics of datasets we use are listed in Table 1. We initialize FORTAP with parameters of the pretrained TUTA. The input is linearized following TUTA by concatenating the text (the prompt built in NRP pretraining task) and the flattened table traversed in row order. Due to memory limit, we only Dataset Enron HiTab DeEx # samples (train/dev/test) 125 k 10 . place (1) header cells, (2) data cells on the same row/column of the formula cell, into the input sequence and skip the other cells. Our input pattern is reasonable as a tradeoff between performance and memory since we find that more than 89% formulas only reference cells on the same row/column. To match different downstream tasks, for the cell with formula, we input its formula token sequence (e.g. (C4-B4)/B4 ) with 40% probability, formula tag [FORMULA] with 30% (the number embedding is set to default) and cell literal value with 30% (e.g. number 42 . 1 ). In experiments, we find it is more effective in Formula MLM to mask either all operators/functions or all referenced cells, so we implement it this way. We first pretrain 400 K steps on sequence length 256 with batch size 32 , and 250 K steps on sequence length 512 with batch size 8 . The whole pretraining phase takes about 4 days on 4 Tesla V100 GPUs. Formula prediction (Chen et al. , 2021a) facilitates spreadsheet end-users by recommending formulas since writing formulas could be time-consuming and error-prone. Given a table and a target cell in table, the task is to predict a formula for the target cell. Formula prediction requires complex in-table numerical reasoning capabilities to predict both referenced cells and involved calculations. Datasets. Enron (Hermans and Murphy-Hill) is a massive database of public Excel Spreadsheet, containing over 17K spreadsheets with rich table structures and formula types. We exclude Enron from our pretraining corpus to prevent data leakage. Tables and formulas are preprocessed in the same way as the pretraining corpus. We divide Enron by sheet and the final dataset contains 100 . 3 K / 12 . 3 K / 12 . 9 K table-formula pairs for train/dev/test. The formula cell in table is regarded as the target cell and the formula is seen as the ground truth in formula prediction task. We follow the evaluation metrics in SpreadsheetCoder (Chen et al. , 2021a): (1) Formula Accu-1154 racy, (2) Sketch Accuracy, (3) Range Accuracy measuring the percentage of correctly predicted formulas, formula sketches (formula using placeholder [RANGE] as referenced cells), and formula ranges (only the referenced cells of formula). Previous to our work, SpreadsheetCoder evaluates formula prediction on collected Google Sheets and Enron. However, we do not directly use its datasets for three reasons: (1) The Google Sheet corpus is not released, and for Enron, SpreadsheetCoder only adopts formulas referencing cells within a limited rectangular neighborhood region ( 21  20 ) of the formula cell, while we argue in real tables the referenced cells can be easily beyond this region. (2) A large proportion of table headers are not properly detected (mentioned in its paper), while we adopt ranges and headers detected by TableSense (Dong et al. , 2019b) and extract table header hierarchies. (3) Despite the inconsistencies above, we try to backtrack the original file to align with SpreadsheetCoder and apply our preprocessing. However, the document IDs of tables in SpreadsheetCoder are mostly empty. Thus, we build our dataset based on Enron and evaluate SpreadsheetCoder on it for a fair comparison. Baselines. We adopt SpreadsheetCoder (Chen et al. , 2021a), TaPEx (Liu et al. , 2021), and TUTA as our baselines. SpreadsheetCoder is a BERT-based model for formula prediction, incorporating headers and contextual information of neighbouring cells of the target cell. TaPEx is a BART-based (Lewis et al. ) table pretraining model, which implicitly learns a SQL executor. Fine-tune. FORTAP consumes all header cells in the table and data cells lying on the same row/column of the target cell just like the manner in pretraining, with a max sequence length, 512 . The [FORMULA] tag is placed at the target cell position in input, whose number embedding is set to default. A two-stage LSTM formula decoder (Dong and Lapata, 2018; Chen et al. , 2021a) accepts the formula cell embedding as input, and generates the formula by first generating formula sketches and then selecting referenced cells. All models in experiments are fine-tuned 800 K steps on Enron. The beam size is 5 for generating formula. Since SpreadsheetCoder only published part of its code, we re-implement it in PyTorch (Paszke et al. , 2019) based on its paper. Appendix B presents details about SpreadsheetCoder. TaPEx is built on BART model and thus naturally supports generation task. We follow the TaPEx table linearization strategy, assign the formula position in the source, and modify the target vocabulary as SpreadsheetCoder (Chen et al. , 2021a) to support generating referenced cells. We use the TaPEx-base model. It is fine-tuned for 30 K steps (converge at about 25 K ) and evaluated on the checkpoint with the best dev performance. Results. Table 2 summarizes the results of formula prediction on the test set. As shown, FORTAP delivers a big improvement over SpreadsheetCoder by +15 . 4% and TaPEx by +12 . 6% on formula accuracy. We deduce that TaPEx falls behind TUTA and FORTAP because (1) the learnt executor may not be suitable for formula prediction, (2) it doesn't leverage hierarchical table structures. FORTAP also outperforms TUTA by +7 . 3% , showing formula pretraining effectively assists formula prediction. We also experiment under a low-resource setting ( 20% training data), and the improvements of FORTAP are more significant, surpassing TUTA by +10 . 2% . Since Enron is not included in our pretraining corpus, this result well indicates formula pretraining can largely benefit formula prediction after seeing large numbers of real formulas. Moreover, we conjecture that formula pretraining potentially improves numerical reasoning capabilities of the model, because the two-stage prediction of formula sketches and ranges relies on numerical calculation and reference capabilities, respectively. Table QA (Pasupat and Liang, 2015; Cheng et al. , 2021) contains a table and an NL question over the table as the model input. Its output can be cell value(s) or number(s) calculated over numerical cell value(s). Table QA calls for both in-table numerical reasoning and table-text joint reasoning. Datasets. There are several datasets (Pasupat and Liang, 2015; Cheng et al. , 2021; Zhu et al. , 2021; Chen et al. , 2021b) focusing on Table QA 1155 or Table-text hybrid QA. We choose to evaluate on HiTab (Cheng et al. , 2021), a hierarchical web table dataset for question answering and data-to-text. First, tables in HiTab contain rich table structures ( 98 . 1% tables are hierarchical) from 29 domains, posing a challenge to numerical reasoning. Second, a large proportion of questions (  40% ) from Statistical Reports demands complex numerical inference over table and text. Moreover, questions in HiTab are revised from sentences written by professional analysts to ensure naturalness and meaningfulness. The QA evaluation metric is Execution Accuracy measuring the percentage of correctly predicted answers. Baselines. We employ TaPas (Herzig et al. , 2020), HiTab model (Cheng et al. , 2021), TaPEx (Liu et al. , 2021), and TUTA as our baselines. TaPas is an end-to-end table parsing model without generating logical forms, which enjoys pretraining on the large-scale table-text corpus from Wikipedia. HiTab devises a hierarchy-aware logical form for hierarchical tables, and predicts the answer using a weakly supervised semantic parser MAPO (Liang et al. , 2018), which is a reinforcement learning framework to systematically explore and generate programs. The question and table are encoded by BERT and the logical forms are generated by an LSTM decoder. TaPEx is introduced in Section 4.2. Fine-tune. We replace the BERT encoder of HiTab model with TUTA and FORTAP , and follow the fine-tuning settings of HiTab. We find that NRP pretrain task under table-text setting mentioned in Section 3 is quite essential for QA performance and thus pretrain 80 , 000 steps more with it on FORTAP in QA before fine-tuning. For TaPEx, we adopt the same table QA strategy in its paper by inputting the table and text as source, and generating the answer as target. The TaPEx-base model is trained for 20 , 000 steps on HiTab. Results. Table 3 summarizes QA results on HiTab. FORTAP achieves SOTA ( 47 . 0% ) using MAPO as the semantic parser, surpassing the best system in HiTab paper with +6 . 3% . Meanwhile, replacing BERT with TUTA does not see a significant performance gain. We conjecture one of the reasons is that TUTA may be not skilled at table-text joint reasoning, and FORTAP enhances this skill by the table-text setting of the NRP task. Finally, FORTAP performs comparatively with TaPEx, a recent pretraining tabular model as a powerful neural SQL executor targeting table reasoning. Note that this (%) Development Test TaPas 39 . result is inspiring since FORTAP is pretrained on spreadsheet tables and can generalize to web table domain (HiTab) with SOTA performance, indicating that the numerical reasoning skills learnt by FORTAP are robust to distinct scenarios. Cell type classification (CTC) (Koci et al. , 2019; Gol et al. , 2019; Gonsior et al. , 2020) aims to interpret tabular data layouts automatically via classifying table cells by their roles in data layouts (e.g., top attribute, data, derived). It requires understanding of table semantics, structures, and numerical relationships considering diverse table layouts. Datasets. DeEx (Koci et al. , 2019) is a widely-studied CTC dataset with tables of various structures and semantics. DeEx includes tables from various domains by mixing three public corpora: Enron (Hermans and Murphy-Hill), Euses (Fisher and Rothermel, 2005), and Fuse(Barik et al. , 2015). Cells in DeEx are categorized into six fine-grained types: metadata, notes, data, left attribute, top attribute , and derived . The evaluation metric is the Macro-F1 score over all cell types. Baselines. We compare FORTAP with two learning-based methods CNNBERT (Dong et al. , 2019a) and Bi-LSTM (Gol et al. , 2019), and three table-pretraining methods TaBERT (Yin et al. , 2020), TaPas (Herzig et al. , 2020), and TUTA. split tables into chunks with a max input sequence length ( 512 ) and distribute headers to each chunk. For cells with formulas, [FORMULA] tags are used as input tokens. We fine-tune 100 epochs on five folds and report the average scores. All these settings are the same as TUTA. Table 4 lists the CTC results on DeEx. FORTAP achieves a SOTA Macro-F1 of 79 . 6% . Specifically, FORTAP largely improves the performance on type derived and notes , surpassing TUTA by 6 . 6% and 7 . 5% . The improvement on derived indicates formula pretraining helps identifying cells derived by calculations over some other cells. Note that derived in DeEx not only includes cells with explicit formulas, but also those cells with hidden (missing) formulas (Koci et al. , 2019), which poses a great challenge to existing methods since it requires discovery of numerical relationships between cells. Thus, this is a strong signal that formula pretraining endows the model with better numerical reasoning capabilities. We think that the improvement on notes mainly benefits from the NRP pretraining task with formula-based prompts as the context, enhancing FORTAP 's capability on table-text joint modeling. In this section, we analyze our method in terms of (1) the effects of different pretraining tasks, (2) whether and to what extent our model learns numerical reasoning skills. Effects of pretraining tasks. We conduct ablation studies on different pretraining tasks on the formula prediction task. Here we pretrain TUTA with each pretraining task and fine-tune on Enron dataset, as summarized in Table 5. We can see that combining all pretraining tasks brings the most gain on formula accuracy. NRP and NCP improve more on range accuracy and sketch accuracy, respectively. This aligns with our design motivation that NRP targets on how to reference and NCP learns how to calculate. To our surprise, Formula MLM alone also largely benefits formula prediction. We deduce the reason is that both MLM and formula prediction requires encoding and recovering/generating capabilities of the formula token sequence. Numerical reasoning skills. We have shown our model learns numerical reasoning skills by two facts: (1) NRP and NCP improve more on the range and sketch accuracy on the formula prediction task, respectively; (2) our model boosts the (%) Formula Sketch Range TUTA 48 . performance of derived cell type on cell type classification. Here we further decompose QA accuracy of different operations on HiTab. The comparison between previous SOTA system BERT(MAPO) and our FORTAP (MAPO) is shown in Table 6. As shown, our model improves most on complex cell selection (cell indexed by  3 headers) and arithmetic (e.g., difference , sum ) problems. Note that complex cell selection not only requires table-text alignment, but also the references between headers considering that mentions of headers in question could be implicit or missing. Meanwhile, our model also handles superlative (e.g., argmax ) and comparative (e.g., less than ) problems better than BERT, despite these types are relatively infrequent in our formula pretraining corpus. To summarize, our model mainly improves numerical skills regarding cell reference and arithmetic, as well as other aspects like comparing and ranking. Table Pretraining. Table pretraining has been widely studied in recent years. Some works mine large-scale table-text pairs as pretraining corpus (Deng et al. , 2020b; Yin et al. , 2020; Herzig et al. , 2020; Wang et al. , 2021b), some leverage annotated table-text datasets (Deng et al. , 2021; Yu et al. , 2020), and some synthesize a table-text corpus by templates (Yu et al. , 2020; Eisensch-los et al. , 2020). Regarding pretraining tasks, they either train the model to recover masked to-kens/column/cell/entity (Yin et al. , 2020; Herzig et al. , 2020; Wang et al. , 2021b; Deng et al. , 2020b), or explicitly learn table-text alignments (Deng et al. , 2021; Yu et al. , 2020). Recently, TaPEx (Liu et al. , 2021) adopts BART (Lewis et al. ) as a neural executor for synthesized SQLs to improve table reasoning. Whereas, our method explores to use real 1157 spreadsheet formulas to guide table pretraining. Numerical reasoning over Natural Language. Numerical reasoning is important in NL domain (Dua et al. , 2019). Numbers even account for 6 . 15% of all unique tokens in English Wikipedia (Thawani et al. , 2021). Various works target improving numerical reasoning skills on NL (Andor et al. , 2019; Geva et al. , 2020; Jin et al. , 2021). Except using pure NL, MathBERT (Peng et al. , 2021) pretrains NL documents with mathematical formulas. In this paper, we target numerical reasoning over (semi-) structured tables. In this paper, we present FORTAP , a numerical-reasoning-aware table pretraining model that learns numerical reasoning capabilities from spreadsheet formulas. Specifically, we design two pretraining tasks to capture numerical reasoning capabilities by explicitly predicting cell reference and calculation relations. Experiments show that FORTAP achieves new SOTA on formula prediction, question answering, and cell type classification. Further analyses indicate that formula pretraining indeed improves numerical reasoning skills of the model. One limitation of FORTAP is that we haven't fully exploit spreadsheet formulas beyond numerical reasoning. For example, logic functions like VLOOKUP and text functions like LEN can be leveraged to guide complex logic and text reasoning, which will be a promising direction in the future. Dataset. Our pretraing corpus is built upon public English spreadsheet files crawled from webs via the search engine (Wang et al. , 2021b), covers various domains, and has been checked by a compliance team in a company to ensure that does not contain sensitive names or uniquely identifies individual people or offensive content. All datasets used for evaluation are licensed public datasets, e.g., for formula prediction, Enron (Hermans and Murphy-Hill) is a public spreadsheet dataset consisting of over 17 K spreadsheet files, and we re-purpose it for formula prediction following (Chen et al. , 2021a). Application. Our model shows its effectiveness in three representative table-related tasks. Formula prediction helps spreadsheet end-users to write formulas which could be tedious and error-prone. Table QA enables users to query on the table without the need of domain background knowledge. Cell type classification assists interpreting fine-grained table semantic structures, which help users to better understand table structures and contents. There may be risks that crooks use tabular models to automatically parse tables/forms to obtain private personal or company data in bulk, which should be prevented.
abstain result objective abstain objective abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other objective other other other other other other abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain