Understanding tables is an important aspect of natural language understanding. Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. Such spurious biases make the model vulnerable to row and column order perturbations. Additionally, prior work has not thoroughly modeled the table structures or table-text alignments, hindering the table-text understanding ability. In this work, we propose a robust and structurally aware table-text encoding architecture TABLEFORMER , where tabular structural biases are incorporated completely through learnable attention biases. TABLEFORMER is (1) strictly invariant to row and column orders, and, (2) could understand tables better due to its tabular inductive biases. Our evaluations showed that TABLEFORMER outperforms strong baselines in all settings on SQA, WTQ and TABFACT table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models' performance drops by 4% 6% when facing such perturbations while TABLEFORMER is not affected. 1 1 Introduction Recently, semi-structured data (e.g. variable length tables without a fixed data schema) has attracted more attention because of its ubiquitous presence on the web. On a wide range of various table reasoning tasks, Transformer based architecture along with pretraining has shown to perform well (Eisen-schlos et al., 2021; Liu et al., 2021). (a) TAPAS predicts incorrect answer based on the original table, while it gives the correct answer if the first row is moved to the end of the table. Liu et al., 2021), where original position ids are used as positional information. Due to the usage of row/column ids and global position ids, prior strategies to linearize table structures introduced spurious row and column order biases (Herzig et al., 2020; Eisenschlos et al., 2020, 2021; Zhang et al., 2020; Yin et al., 2020). Therefore, those models are vulnerable to row or column order perturbations. But, ideally, the model should make consistent predictions regardless of the row or column ordering for all practical purposes. For instance, in Figure 1, the predicted answer of TAPAS model (Herzig et al., 2020) for Question (a) Of all song lengths, which one is the longest? based on the original table is 5:00 , which is incorrect. However, if the first row is adjusted to the end of the table during inference, the model gives the correct length 5:02 as an-528 swer. This probing example shows that the model being aware of row order information is inclined to select length values to the end of the table due to spurious training data bias. In our experiments on the SQA dataset, TAPAS models exhibit a 4% 6% (Section 5.2) absolute performance drop when facing such answer-invariant perturbations. Besides, most prior work (Chen et al., 2020; Yin et al., 2020) did not incorporate enough structural biases to models to address the limitation of sequential Transformer architecture, while others inductive biases which are either too strict (Zhang et al., 2020; Eisenschlos et al., 2021) or computationally expensive (Yin et al., 2020). To this end, we propose TABLEFORMER , a Transformer architecture that is robust to row and column order perturbations, by incorporating structural biases more naturally. TABLEFORMER relies on 13 types of task-independent table  text attention biases that respect the table structure and table-text relations. For Question (a) in Figure 1, TABLEFORMER could predict the correct answer regardless of perturbation, because the model could identify the same row information with our same row bias, avoiding spurious biases introduced by row and global positional embeddings. For Question (b), TAPAS predicted only partially correct answer, while TABLEFORMER could correctly predict Spain, Ukraine as answers. That's because our cell to sentence bias could help table cells ground to the paired sentence. Detailed attention bias types are discussed in Section 5.2. Experiments on 3 table reasoning datasets show that TABLEFORMER consistently outperforms original TAPAS in all pretraining and intermediate pretraining settings with fewer parameters. Also, TABLEFORMER 's invariance to row and column perturbations, leads to even larger improvement over those strong baselines when tested on perturbations. Our contributions are as follows:  We identified the limitation of current table-text encoding models when facing row or column perturbation. We propose TABLEFORMER , which is guaranteed to be invariant to row and column order perturbations, unlike current models. TABLEFORMER encodes table-text structures better, leading to SoTA performance on SQA dataset, and ablation studies show the effectiveness of the introduced inductive biases. In this section, we discuss TAPAS which serves as the backbone of the recent state-of-the-art table-text encoding architectures. TAPAS (Herzig et al., 2020) uses Transformer architecture in a BERT like fashion to pretrain and finetune on tabular data for table-text understanding tasks. This is achieved by using linearized table and texts for masked language model pre-training. In the fine-tuning stage, texts in the linearized table and text pairs are queries or statements in table QA or table-text entailment tasks, respectively. Specifically, TAPAS uses the tokenized and flat-tened text and table as input, separated by [SEP] token, and prefixed by [CLS] . Besides token, segment, and global positional embedding introduced in BERT (Devlin et al., 2019), it also uses rank embedding for better numerical understanding. Moreover, it uses column and row embedding to encode table structures. Concretely, for any table-text linearized sequence S = { v 1 , v 2 ,    , v n } , where n is the length of table-text sequence, the input to TAPAS is summation of embedding of the following: token ids ( W ) = { w v 1 , w v 2 ,    , w v n } positional ids ( B ) = { b 1 , b 2 ,    , b n } segment ids ( G ) = { g seg 1 , g seg 2 ,    , g seg n } column ids ( C ) = { c col 1 , c col 2 ,    , c col n } row ids ( R ) = { r row 1 , r row 2 ,    , r row n } rank ids ( Z ) = { z rank 1 , z rank 2 ,    , z rank n } where seg i , col i , row i , rank i correspond to the segment, column, row, and rank id for the i th token, respectively. As for the model, TAPAS uses BERT's self-attention architecture (Vaswani et al., 2017) off-the-shelf. Each Transformer layer includes a multihead self-attention sub-layer, where each token attends to all the tokens. Let the layer input H = [ h 1 , h 2 ,    , h n ] (cid:62)  R n  d corresponding to S , where d is the hidden dimension, and h i  R d  1 is the hidden representation at position i . For a single-head self-attention sub-layer, the input H is projected by three matrices WQ  R d  d K , WK  R d  d K , and WV  R d  d V to the corresponding representations Q , K , and V : Q = HWQ , V = HWV , K = HWK (1) 529 Transformer (Self Attention) Transformer (Self Attention)                                                                                                                          Learnable structure enforced attention bias scalars . Then, the output of this single-head self-attention sub-layer is calculated as: Attn ( H ) = softmax ( QK (cid:62)  d K ) V (2) 3 TABLEFORMER : Robust Structural Table Encoding As shown in Figure 2, TABLEFORMER encodes the general table structure along with the associated text by introducing task-independent relative attention biases for table-text encoding to facilitate the following: (a) structural inductive bias for better table understanding and table-text alignment, (b) robustness to table row/column perturbation. Input of TABLEFORMER . TABLEFORMER uses the same token embeddings W , segment embeddings G , and rank embeddings Z as TAPAS . However, we make 2 major modifications: 1) No row or column ids. We do not use row embeddings R or column embeddings C to avoid any potential spurious row and column order biases. 2) Per cell positional ids. To further remove any inter-cell order information, global positional embeddings B are replaced by per cell positional embeddings P = { p pos 1 , p pos 2 ,    , p pos n } , where we follow Eisenschlos et al. (2021) to reset the index of positional embeddings at the beginning of each cell, and pos i correspond to the per cell positional id for the i th token. Positional Encoding in TABLEFORMER . Note that the Transformer model either needs to specify different positions in the input (i.e. absolute positional encoding of Vaswani et al. (2017)) or encode the positional dependency in the layers (i.e. relative positional encoding of Shaw et al. (2018)). TABLEFORMER does not consume any sort of column and row order information in the input. The main intuition is that, for cells in the table, the only useful positional information is whether two cells are in the same row or column and the column header of each cell, instead of the absolute order of the row and column containing them. Thus, inspired by relative positional encoding (Shaw et al., 2018) and graph encoding (Ying et al., 2021), we capture this with a same column/row relation as one kind of relative position between two linearized tokens. Similarly, we uses 12 such table-text structure relevant relations (including same cell, cell to header and so on) and one extra type representing all other relations not explicitly defined. All of them are introduced in the form of learnable 530 attention bias scalars. Formally, we consider a function  ( v i , v j ) : V  V  N , which measures the relation between v i and v j in the sequence ( v i , v j  S ). The function  can be defined by any relations between the tokens in the table-text pair. Attention Biases in TABLEFORMER . In our work,  ( v i , v j ) is chosen from 13 bias types, corresponding to 13 table-text structural biases. The attention biases are applicable to any table-text pair and can be used for any downstream task:  same row identifies the same row information without ordered row id embedding or global positional embedding, which help the model to be invariant to row perturbations,  same column , header to column cell , and cell to column header incorporates the same column information without ordered column id embedding,  cell to column header makes each cell aware of its column header without repeated column header as features,  header to sentence and cell to sentence help column grounding and cell grounding of the paired text,  sentence to header , sentence to cell , and sentence to sentence helps to understand the sentence with the table as context,  header to same header and header to other header for better understanding of table schema, and same cell bias for cell content understanding. We assign each bias type a learnable scalar, which will serve as a bias term in the self-attention module. Specifically, each self-attention head in each layer have a set of learnable scalars { b 1 , b 2 ,    , b 13 } corresponding to all types of introduced biases. For one head in one self-attention sub-layer of TABLEFORMER , Equation 2 in the Transformer is replaced by:  A = QK (cid:62)  d K , A =  A +  A (3) Attn ( H ) = softmax ( A ) V (4) where  A is a matrix capturing the similarity between queries and keys,  A is the Attention Bias Matrix, and  A i,j = b  ( v i ,v j ) . Relation between TABLEFORMER and ETC. ETC (Ainslie et al., 2020) uses vectors to represent relative position labels, although not directly applied to table-text pairs due to its large computational overhead (Eisenschlos et al., 2021). TABLEFORMER differs from ETC in the following aspects (1) ETC uses relative positional embeddings while TABLEFORMER uses attention bias scalars. In practice, we observed that using relative positional embeddings increases training time by more than 7x, (2) ETC uses global memory and local attention, while TABLEFORMER uses pairwise attention without any global memory overhead, (3) ETC uses local sparse attention with masking, limiting its ability to attend to all tokens, (4) ETC did not explore table-text attention bias types exhaustively. Another table encoding model MATE (Eisensch-los et al., 2021) is vulnerable to row and column perturbations, and shares limitation (3) and (4). Table Question Answering. For the table QA task, we conducted experiments on WikiTableQues-tions (WTQ) (Pasupat and Liang, 2015) and Sequential QA (SQA) (Iyyer et al., 2017) datasets. WTQ was crowd-sourced based on complex questions on Wikipedia tables. SQA is composed of 6 , 066 question sequences (2.9 question per sequence on average), constructed by decomposing a subset of highly compositional WTQ questions. Table-Text Entailment. For the table-text entailment task, we used TABFACT dataset (Chen et al., 2020), where the tables were extracted from Wikipedia and the sentences were written by crowd workers. Among total 118, 000 sentences, each one is a positive (entailed) or negative sentence. Perturbation Evaluation Set. For SQA and TABFACT , we also created new test sets to measure models' robustness to answer-invariant row and column perturbations during inference. Specifically, 531 row and column orders are randomly perturbed for all tables in the standard test sets. Pre-training All the models are first tuned on the Wikipidia text-table pretraining dataset (Herzig et al., 2020), optionally tuned on synthetic dataset at an intermediate stage (inter) (Eisenschlos et al., 2020), and finally fine-tuned on the target dataset. To get better performance on WTQ, we follow Herzig et al. (2020) to further pretrain on SQA dataset after the intermediate pretraining stage in the inter-sqa setting. Evaluation For SQA, we report the cell selection accuracy for all questions (ALL) using the official evaluation script, cell selection accuracy for all sequences (SEQ), and the denotation accuracy for all questions (ALL d ). To evaluate the models' robustness in the instance level after perturbations, we also report a lower bound of example prediction variation percentage: V P = ( t2f + f2t ) ( t2t + t2f + f2t + f2f ) (5) where t2t, t2f, f2t, and f2f represents how many example predictions turning from correct to correct, from correct to incorrect, from incorrect to correct and from incorrect to incorrect, respectively, after perturbation. We report denotation accuracy on WTQ and binary classification accuracy on TABFACT respectively. We use TAPASBASE and TAPASLARGE as baselines, where Transformer architectures are exactly same as BERTBASE and BERTLARGE (Devlin et al., 2019), and parameters are initialized from BERTBASE and BERTLARGE respectively. Correspondingly, we have our TABLEFORMERBASE and TABLEFORMERLARGE , where attention bias scalars are initialized to zero, and all other parameters are initialized from BERTBASE and BERTLARGE . Could we alleviate the spurious ordering biases by data augmentation alone, without making any modeling changes? To answer this, we train another set of models by augmenting the training data 2 We fixed perturbation random seeds to make our results reproducible. For each table in the training set, we randomly shuffle all rows and columns (including corresponding column headers), creating a new table with the same content but different orders of rows and columns. Multiple perturbed versions of the same table were created by repeating this process { 1 , 2 , 4 , 8 , 16 } times with different random seeds. For table QA tasks, selected cell positions are also adjusted as final answers according to the perturbed table. The perturbed table-text pairs are then used to augment the data used to train the model. During training, the model takes data created by one spe-cific random seed in one epoch in a cyclic manner. How robust are existing (near) state-of-the-art table-text encoding models to semantic preserving perturbations in the input? How does TABLEFORMER compare with existing table-text encoding models when tested on similar perturbations, both in terms of performance and robustness? 3 By perturbation, we mean shuffling row and columns instead of changing/swapping content blindly. Can we use perturbation based data augmentation to achieve robustness at test time? Which attention biases in TABLEFORMER contribute the most to performance? Table 1, 2, and 3 shows TABLEFORMER performance on SQA, TABFACT , and WTQ, respectively. As can be seen, TABLEFORMER outperforms corresponding TAPAS baseline models in all settings on SQA and WTQ datasets, which shows the general effectiveness of TABLEFORMER 's structural biases in Table QA datasets. Specifi-cally, TABLEFORMERLARGE combined with intermediate pretraining achieves new state-of-the-art performance on SQA dataset. Similarly, Table 2 shows that TABLEFORMER also outperforms TAPAS baseline models in all settings, which shows the effectiveness of TABLEFORMER in the table entailment task. Note that, Liu et al. (2021) is not comparable to our results, because they used different pretraining data, different pretraining objectives, and BART NLG model instead of BERT NLU model. But TABLEFORMER attention bias is compatible with BART model. One of our major contributions is to systematically evaluate models' performance when facing row and column order perturbation in the testing stage. Ideally, model predictions should be consistent on table QA and entailment tasks when facing such perturbation, because the table semantics remains the same after perturbation. However, in Table 1 and 2, we can see that in our perturbed test set, performance of all TAPAS models drops significantly in both tasks. TAPAS models drops by at least 3.7% and up to 6.5% in all settings on SQA dataset in terms of ALL accuracy, while our TABLEFORMER being strictly invariant to such row and column order perturbation leads to no drop in performance. 4 Thus, in the perturbation setting, TABLEFORMER outperforms all TAPAS baselines even more significantly, with at least 6.2% and 2.4% improvements on SQA and TABFACT dataset, respectively. In the instance level, we can see that, with TAPAS , there are many example predictions changed due to high V P , while there is nearly no example predictions changed with TABLEFORMER (around zero V P ). 4 In SQA dataset, there is at most absolute 0.1% performance drop because of some bad data point issues. Specifically, some columns in certain tables are exactly the same, but the ground-truth selected cells are in only one of such columns. TABLEFORMER would select from one column randomly. We compare the model sizes of TABLEFORMER and TAPAS in Table 4. We added only a few attention bias scalar parameters (13 parameters per head per layer) in TABLEFORMER , which is negligible compared with the BERT model size. Meanwhile, we delete two large embedding metrics (512 row ids and 512 column ids). Thus, TABLEFORMER outperforms TAPAS with fewer parameters. In this section, we experiment with several variants of TABLEFORMER to understand the effectiveness of its submodules. The performance of all variants of TAPAS and TABLEFORMER that we tried on the SQA development set is shown in Table 5. Learnable Attention Biases v/s Masking. Instead of adding learnable bias scalars, we mask out some attention scores to restrict attention to those tokens in the same columns and rows, as well as the paired sentence, similar to Zhang et al. (2020) (SAT). We can see that TAPASBASE-SAT performs worse than TAPASBASE , which means that restricting attention to only same columns and rows by masking reduce the modeling capacity. This led to choosing soft bias addition over hard masking. Attention Bias Scaling. Unlike TABLEFORMER , we also tried to add attention biases before the scaling operation in the self-attention module (SO). Specifically, we compute pair-wise attention score by: A ij = ( h (cid:62) i WQ )( h (cid:62) j WK ) (cid:62) +  A ij  d K (6) instead of using: A ij = ( h (cid:62) i WQ )( h (cid:62) j WK ) (cid:62)  d K +  A ij , (7) rc-gp c-gp gp pcp TAPASBASE 57.6 47.4 46.4 29.1 TAPASBASE-SAT 45.2 -TABLEFORMERBASE-SO 60.0 60.2 59.8 60.7 TABLEFORMERBASE 62.2 61.5 61.7 61.9 Table 5: ALL questions' cell selection accuracy of TABLEFORMER variants on SQA development set. which is the element-wise version of Equation 1 and 3. However, Table 5 shows that TABLEFORMERBASE-SO performs worse than TABLEFORMERBASE , showing the necessity of adding attention biases after the scaling operation. We think the reason is that the attention bias term does not require scaling, because attention bias scalar magnitude is independent of d K , while the dot products grow large in magnitude for large values of d K . Thus, such bias term could play an more important role without scaling, which helps each attention head know clearly what to pay more attention to according to stronger inductive biases. Row, Column, & Global Positional IDs. With TAPASBASE , TABLEFORMERBASE-SO , and TABLEFORMERBASE , we first tried the full-version where row ids, column ids, and global positional ids exist as input ( rc-gp ). Then, we deleted row ids ( c-gp ), and column ids ( gp ) sequentially. Finally, we changed global positional ids in gp to per-cell positional ids ( pcp ). Table 5 shows that TAPASBASE performs significantly worse from rc-gp  c-gp  gp  pcp , because table structure information are deleted sequentially during such process. However, with TABLEFORMERBASE , there is no obvious performance drop during the same process. That shows the structural inductive biases in TABLEFORMER can provide complete table structure information. Thus, row ids, column ids and global positional ids are not necessary in TABLEFORMER . We pick TABLEFORMER pcp setting as our final version to conduct all other experiments in this paper. In this way, TABLEFORMER is strictly invariant to row and column order perturbation by avoiding spurious biases in those original ids. As stated in Section 4.3, perturbing row and column orders as augmented data during training can serve as another possible solution to alleviate the spurious row/column ids bias. Table 6 shows the performance of TABPASBASE model trained with additional {1, 2, 4, 8, 16} perturbed versions of each table as augmented data. We can see that the performance of TAPASBASE on SQA dataset improves with such augmentation. Also, as the number of perturbed versions of each table increases, model performance first increases and then decreases, reaching the best results with 8 perturbed versions. We suspect that too many versions of the same table confuse the model about different row and column ids for the same table, leading to decreased performance from 8p to 16p. Despite its usefulness, such data perturbation is still worse than TABLEFORMER , because it could not incorporate other relevant text-table structural inductive biases like TABLEFORMER . Although, such data augmentation makes the model more robust to row and column order perturbation with smaller V P compared to standard TAPASBASE , there is still a significant prediction drift after perturbation. As shown in Table 6, V P decreases from 1p to 16p, however, the best V P (7.0%) is still much higher than (nearly) no variation (0.1%) of TABLEFORMER . To sum up, TABLEFORMER is superior to row and column order perturbation augmentation, because of its additional structural biases and strictly consistent predictions after perturbation. We conduct ablation study to demonstrate the utility of all 12 types of defined attention biases. For each ablation, we set the corresponding attention bias type id to others bias id. Table 7 shows TAPASBASE 's performance SQA dev set. Overall, all types of attention biases help the TABLEFORMER performance to some extent, due to certain performance drop after deleting each bias type. Amongst all the attention biases, deleting same row bias leads to most significant performance drop, showing its crucial role for encoding table row structures. There is little performance drop after deleting same column bias, that's because TABLEFORMER could still infer the same column information through cell to its column header and header to its column cell biases. After deleting all same column information ( same column , cell to column header and header to column cell biases), TABLEFORMER performs significantly worse without encoding column structures. Similarly, there is little performance drop after deleting same cell bias, because TABLEFORMER can still infer same cell information through same row and same column biases. TABLEFORMERTABLEFORMER increases the training time by around 20% , which might not be ideal for very long tables and would require a scoped approach. Secondly, with the strict row and column order invariant property, TABLEFORMER cannot deal with questions based on absolute orders of rows in tables. This however is not a practical requirement based on the current dataset. Doing a manual study of 1800 questions in SQA dataset, we found that 535 there are 4 questions 5 (0.2% percentage) whose answers depend on orders of rows. Three of them asked which one is at the top of the table , another asks which one is listed first . However, these questions could be potentially answered by adding back row and column order information based on TABLEFORMER . Transformers for Tabular Data. Yin et al. (2020) prepended corresponding column headers to cells contents, and Chen et al. (2020) used corresponding column headers as features for cells. However, such methods encode each table header multiple times, leading to duplicated computing overhead. Also, tabular structures (e.g. same row information) are not fully incorporated to such models. Meanwhile, Yin et al. (2020) leveraged row encoder and column encoder sequentially, which introduced much computational overhead, thus requiring retrieving some rows as a preprocessing step. Finally, SAT (Zhang et al., 2020), Deng et al. (2021) and Wang et al. (2021) restricted attention to same row or columns with attention mask, where such inductive bias is too strict that cells could not directly attend to those cells in different row and columns, hindering the modeling ability according to Table 5. Liu et al. (2021) used the seq2seq BART generation model with a standard Transformer encoder-decoder architecture. In all models mentioned above, spurious inter-cell order biases still exist due to global positional ids of Transformer, leading to the vulnerability to row or column order perturbations, while our TABLEFORMER could avoid such problem. Mueller et al. (2019) and Wang et al. (2020) also used relative positional encoding to encode table structures, but they modeled the relations as learnable relation vectors, whose large overhead prevented pretraining and led to poor performance without pretraining, similarly to ETC (Ainslie et al., 2020) explained in Section 3. Structural and Relative Attention. Modified attention scores has been used to model relative positions (Shaw et al., 2018), long documents (Dai et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020), and graphs (Ying et al., 2021). But adding 5 We find such 4 questions by manually looking at all 125 questions where the model predictions turn from correct to incorrect after replacing TAPASLARGE with TABLEFORMERLARGE . In this paper, we identified the vulnerability of prior table encoding models along two axes: (a) capturing the structural bias, and (b) robustness to row and column perturbations. To tackle this, we propose TABLEFORMER , where learnable task-independent learnable structural attention biases are introduced, while making it invariant to row/column order at the same time. Experimental results showed that TABLEFORMER outperforms strong baselines in 3 table reasoning tasks, achieving state-of-the-art performance on SQA dataset, especially when facing row and column order perturbations, because of its invariance to row and column orders. We thank Julian Eisenschlos, Ankur Parikh, and the anonymous reviewers for their feedbacks in improving this paper. The authors foresee no ethical concerns with the research presented in this paper.
abstain abstain abstain abstain objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other other abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain method other abstain abstain other abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other other other other other other other abstain other other other abstain method abstain abstain objective abstain other method