Sequence modeling has demonstrated state-of-the-art performance on natural language and document understanding tasks. However, it is challenging to correctly serialize tokens in form-like documents in practice due to their variety of layout patterns. We propose FormNet, a structure-aware sequence model to mitigate the suboptimal serialization of forms. First, we design Rich Attention that leverages the spatial relationship between tokens in a form for more precise attention score calculation. Second, we construct Super-Tokens for each word by embedding representations from their neighboring tokens through graph convolutions. FormNet therefore explicitly recovers local syntactic information that may have been lost during serialization. In experiments, FormNet outperforms existing methods with a more compact model size and less pretraining data, establishing new state-of-the-art performance on CORD, FUNSD and Payment benchmarks. Form-like document understanding is a surging research topic because of its practical applications in automating the process of extracting and organizing valuable text data sources such as marketing documents, advertisements and receipts. Typical documents are represented using natural languages; understanding articles or web content (Antonacopoulos et al., 2009; Luong et al., 2012; Soto and Yoo, 2019) has been studied extensively. However, form-like documents often have more complex layouts that contain structured objects, such as tables and columns. Therefore, form documents have unique challenges compared to natural language documents stemming from their structural characteristics, and have been largely under-explored. In this work, we study critical information extraction from form documents, which is the funda-(cid:79)(cid:213)(cid:341) (cid:156)(cid:177)(cid:259)(cid:315)(cid:213) (cid:203)(cid:271)(cid:264)(cid:296)(cid:177)(cid:265)(cid:341)(cid:1085)(cid:265)(cid:177)(cid:264)(cid:213) (cid:119)(cid:122)(cid:38)(cid:88)(cid:132)(cid:65)(cid:28)(cid:38)(cid:1109)(cid:62)(cid:1)(cid:81)(cid:81)(cid:1109)(cid:28)(cid:94)(cid:122)(cid:119)(cid:94)(cid:122)(cid:1)(cid:132)(cid:38)(cid:1109)(cid:126)(cid:38)(cid:122)(cid:156)(cid:65)(cid:28)(cid:38)(cid:126) (cid:303)(cid:213)(cid:265)(cid:310)(cid:1085)(cid:209)(cid:177)(cid:310)(cid:213) (cid:938)(cid:939)(cid:1080)(cid:938)(cid:940)(cid:1080)(cid:945)(cid:946) (cid:299)(cid:213)(cid:203)(cid:213)(cid:241)(cid:334)(cid:213)(cid:209)(cid:1085)(cid:209)(cid:177)(cid:310)(cid:213) (cid:34)(cid:213)(cid:203)(cid:1109)(cid:938)(cid:941)(cid:1109)(cid:945)(cid:946) (cid:299)(cid:213)(cid:203)(cid:213)(cid:241)(cid:334)(cid:213)(cid:299)(cid:1085)(cid:177)(cid:209)(cid:209)(cid:299)(cid:213)(cid:303)(cid:303) (cid:81)(cid:94)(cid:122)(cid:65)(cid:81)(cid:81)(cid:1)(cid:122)(cid:34)(cid:1045)(cid:1109)(cid:65)(cid:88)(cid:28)(cid:1044)(cid:1109)(cid:94)(cid:88)(cid:38)(cid:1109)(cid:119)(cid:1)(cid:122)(cid:79)(cid:1109)(cid:1)(cid:156)(cid:38)(cid:88)(cid:137)(cid:38)(cid:1109)(cid:938)(cid:945)(cid:132)(cid:62)(cid:1109)(cid:56)(cid:81)(cid:1109)(cid:88)(cid:38)(cid:157)(cid:1109)(cid:163)(cid:94)(cid:122)(cid:79)(cid:1045)(cid:1109)(cid:88)(cid:1044)(cid:1109)(cid:163)(cid:1044)(cid:1109)(cid:938)(cid:937)(cid:937)(cid:938)(cid:943) Figure 1: An illustration of the form document information extraction task. mental subtask of form document understanding. Following the success of sequence modeling in natural language understanding (NLU), a natural approach to tackle this problem is to first serialize the form documents and then apply state-of-the-art sequence models to them. For example, Palm et al. (2017) use Seq2seq (Sutskever et al., 2014) with RNN, and Hwang et al. (2019) use transformers (Vaswani et al., 2017). However, interwoven columns, tables, and text blocks make serialization difficult, substantially limiting the performance of a strict serialization approach. To model the structural information present in documents, Katti et al. (2018); Zhao et al. (2019); Denk and Reisswig (2019) treat the documents as 2D image inputs and directly apply convolutional networks on them to preserve the spatial context during learning and inference. However, the performance is limited by the resolution of the 2D input grids. Another approach is a two-step pipeline (Hi-rano et al., 2007) that leverages computer vision algorithms to first infer the layout structures of forms and then perform sequence information extraction. The methods are mostly demonstrated on plain text articles or documents (Yang et al., 2017; Soto and Yoo, 2019) but not on highly entangled form documents (Davis et al., 2019; Zhang et al., 2019). In this work, we propose FormNet, a structure-aware sequence model to mitigate the suboptimal 3735 (cid:55)(cid:36)(cid:53)(cid:3)(cid:49)(cid:44)(cid:38)(cid:3)(cid:48)(cid:50)(cid:44)(cid:54)(cid:55)(cid:3)(cid:48)(cid:40)(cid:49)(cid:55)(cid:3)(cid:46)(cid:50)(cid:50)(cid:47)(cid:3)(cid:47)(cid:76)(cid:74)(cid:75)(cid:87)(cid:86)(cid:3)(cid:46)(cid:54)(cid:3)(cid:3)(cid:16)(cid:3)(cid:3) (cid:90)(cid:75)(cid:76)(cid:87)(cid:72) (cid:3)(cid:87)(cid:76)(cid:83)(cid:16)(cid:3)(cid:3)(cid:28)(cid:17)(cid:20)(cid:3)(cid:17)(cid:27)(cid:27)(cid:3)(cid:20)(cid:23)(cid:17)(cid:19)(cid:3)(cid:17)(cid:23)(cid:26)(cid:23)(cid:3)(cid:83)(cid:76)(cid:81)(cid:74)(cid:3)(cid:80)(cid:68)(cid:86)(cid:78)(cid:72)(cid:71) (cid:41)(cid:135)(cid:31)(cid:3)(cid:84)(cid:306)(cid:238)(cid:212)(cid:294)(cid:3)(cid:125)(cid:212)(cid:244)(cid:276)(cid:350)(cid:338) (cid:125)(cid:276)(cid:238)(cid:273)(cid:3)(cid:4)(cid:458)(cid:248)(cid:300)(cid:345)(cid:276)(cid:306)(cid:300) (cid:129)(cid:238)(cid:306)(cid:334)(cid:248)(cid:3)(cid:4)(cid:244)(cid:288)(cid:350)(cid:338)(cid:345)(cid:299)(cid:248)(cid:300)(cid:345) (cid:533)(cid:16)(cid:54)(cid:78)(cid:72)(cid:79)(cid:72)(cid:87)(cid:82)(cid:81)(cid:3)(cid:42)(cid:85)(cid:68)(cid:83)(cid:75)(cid:3)(cid:82)(cid:81)(cid:3)(cid:55)(cid:82)(cid:78)(cid:72)(cid:81)(cid:86) (cid:858)(cid:212)(cid:859)(cid:3)(cid:129)(cid:212)(cid:299)(cid:331)(cid:294)(cid:248)(cid:3)(cid:37)(cid:306)(cid:238)(cid:350)(cid:299)(cid:248)(cid:300)(cid:345)(cid:3)(cid:68)(cid:300)(cid:331)(cid:350)(cid:345) (cid:858)(cid:237)(cid:859)(cid:3)(cid:135)(cid:248)(cid:375)(cid:345)(cid:3)(cid:129)(cid:248)(cid:334)(cid:276)(cid:212)(cid:294)(cid:276)(cid:384)(cid:212)(cid:345)(cid:276)(cid:306)(cid:300)(cid:3)(cid:370)(cid:276)(cid:345)(cid:273)(cid:3)(cid:858)(cid:237)(cid:859)(cid:3)(cid:41)(cid:135)(cid:31)(cid:3)(cid:970)(cid:3)(cid:125)(cid:276)(cid:238)(cid:273)(cid:3)(cid:4)(cid:458)(cid:248)(cid:300)(cid:345)(cid:276)(cid:306)(cid:300) (cid:858)(cid:238)(cid:859)(cid:3)(cid:129)(cid:350)(cid:331)(cid:248)(cid:334)(cid:870)(cid:135)(cid:306)(cid:291)(cid:248)(cid:300)(cid:338)(cid:3)(cid:237)(cid:376)(cid:858)(cid:238)(cid:859)(cid:3)(cid:60)(cid:334)(cid:212)(cid:331)(cid:273)(cid:3)(cid:84)(cid:248)(cid:212)(cid:334)(cid:300)(cid:276)(cid:300)(cid:268) Figure 2: A walk-through example of the proposed Rich Attention and Super-Tokens of FormNet. serialization of forms by bridging the gap between plain sequence models and grid-like convolutional models. Specifically, we first design Rich Attention , which leverages the spatial relationships between tokens in a form to calculate a more structurally meaningful attention score, and apply it in a recent transformer architecture for long documents (Ainslie et al., 2020). Second, we construct Super-Tokens for each word in a form by embedding representations from their neighboring tokens through graph convolutions. The graph construction process leverages strong inductive biases about how tokens are related to one another spatially in forms. Essentially, given a form document, FormNet builds contextualized Super-Tokens before serialization errors can be propagated. A transformer model then takes these Super-Tokens as input to perform sequential entity tagging and extraction. In our experiments, FormNet outperforms existing methods while using (1) smaller model sizes and (2) less pre-training data while (3) avoiding the need for vision features. In particular, FormNet achieves new best F1 scores on CORD and FUNSD (97.28% and 84.69%, respectively) while using a 64% sized model and 7.1x less pre-training data than the most recent DocFormer (Appalaraju et al., 2021). Document information extraction was first studied in handcrafted rule-based models (Lebourgeois et al., 1992; O'Gorman, 1993; Ha et al., 1995; Simon et al., 1997). Later Marinai et al. (2005); Shilman et al. (2005); Wei et al. (2013); Chiticariu et al. (2013); Schuster et al. (2013) use learning-based approaches with engineered features. These methods encode low-level raw pixels (Marinai et al., 2005) or assume form templates are known a priori (Chiticariu et al., 2013; Schuster et al., 2013), which limits their generalization to documents with specific layout structures. In addition to models with limited or no learning capabilities, neural models have also been studied. Palm et al. (2017); Aggarwal et al. (2020) use an RNN for document information extraction, while Katti et al. (2018); Zhao et al. (2019); Denk and Reisswig (2019) investigate convolutional models. There are also self-attention networks (transformers) for document information extraction, motivated by their success in conventional NLU tasks. Majumder et al. (2020) extend BERT to representation learning for form documents. Gar-ncarek et al. (2020) modified the attention mechanism in RoBERTa (Liu et al., 2019b). Xu et al. (2020, 2021); Powalski et al. (2021); Appalaraju et al. (2021) are multimodal models that combine BERT-like architectures (Devlin et al., 2019) and advanced computer vision models to extract visual content in images. Similarly, SPADE (Hwang et al., 2021) is a graph decoder built upon the transformer models for better structure prediction compared to simple BIO tagging. The proposed Form-3736 t 0 t 2 t 1 t 3 t 4 t 5 t 7 t 6 v 0 v 2 v 1 v 3 v 4 v 5 v 7 v 6 Input document with tokens t k returned by an OCR engine Graph construction + Message passing Super-tokens by graph embedding ETC w/ RichA tt Token embedding to entity BIOES logits v 0 v 2 v 1 v 3 v 4 v 5 v 7 v 6 GCN Viterbi B 2 E 2 I 2 O O S 5 E 9 B 9 v 0 v 2 v 1 v 3 v 4 v 5 v 7 v 6 Entity BIOES logits to BOIES decoding Token serialization Entity Class 2 Outside Entity Class 5 Entity Class 9 Figure 3: System overview of the proposed FormNet for form document key information extraction. Net is orthogonal to multimodal transformers and SPADE. Compared with multimodal models, FormNet focuses on modeling relations between words through graph convolutional learning as well as Rich Attention without using any visual modality; compared with SPADE, FormNet uses a graph encoder to encode inductive biases in form input. A straightforward extension would be to combine FormNet with either layout transformers or SPADE for capturing visual cues or better decoding, which we leave for future work. Graph learning with sequence models has also been studied. On top of the encoded information through graph learning, Qian et al. (2019); Liu et al. (2019a); Yu et al. (2020) use RNN and CRF while we study Rich Attention in FormNet for decoding. Peng et al. (2017); Song et al. (2018) do not study document information extraction. Problem Formulation. Given serialized 1 words of a form document, we formulate the problem as sequential tagging for tokenized words by predicting the corresponding key entity classes for each token. Specifically, we use the BIOES scheme  { Begin, Inside, Outside, End, Single } (Ratinov and Roth, 2009) to mark the spans of entities in token sequences and then apply the Viterbi algorithm. Proposed Approach. By treating the problem as a sequential tagging task after serialization, we can adopt any sequence model. To handle potentially long documents (e.g. multi-page documents), 1 Different Optical Character Recognition (OCR) engines implement different heuristics. In practice, it is common to see an entity sequence cross multiple spans of a form document, demonstrating the difficulty of recovering from serialization errors. As illustrated in Figure 2(b), 9.1 is next to tip, while ping masked belong to the same entity as tipare distant from it under the imperfect serialization. Our remedy is to encode the original 2D structural patterns of forms in addition to positions within the serialized sentences. We propose two novel components to enhance ETC: Rich Attention and Super-Tokens (Figure 2). Rich Attention captures not only the semantic relationship but also the spatial distance between every pair of tokens in ETC's attention component. Super-tokens are constructed by graph convolutional networks before being fed into ETC. They model local relationships between pairs of tokens that might not be visible to each other or correctly inferred in an ETC model after suboptimal serialization. Figure 3 shows the overall system pipeline. We discuss the details of ETC in Sec. 3.1, Rich Attention in Sec. 3.2, and Super-Token in Sec. 3.3. Transformers (Vaswani et al., 2017) have demonstrated state-of-the-art performance on sequence modeling compared with RNNs. Extended Transformer Construction (ETC; Ainslie et al., 2020) further scales transformers to long sequences by replacing standard (quadratic complexity) atten-2 One can replace ETC with other long-sequence models, such as Zaheer et al. (2020). tion with a sparse global-local attention mechanism. The small number of dummy global tokens attend to all input tokens, but the input tokens attend only locally to other input tokens within a spec-ified local radius. An example can be found in Figure 2(b). As a result, space and time complexity are linear in the long input length for a fixed local radius and global input length. Furthermore, ETC allows a specialized implementation for efficient computation under this design. We refer interested readers to Ainslie et al. (2020) for more details. In this work, we adopt ETC with a single global token as the backbone, as its linear complexity of attention with efficient implementation is critical to long document modeling in practice (e.g. thousands of tokens per document). A key component in transformers for sequence modeling is the positional encoding (Vaswani et al., 2017), which models the positional information of each token in the sequence. Similarly, the original implementation of ETC uses Shaw et al. (2018) for (relative) positional encoding. However, token offsets measured based on the error-prone serialization may limit the power of positional encoding. We address this inadequacy by proposing Rich Attention as an alternative, discussed in Section 3.2. Approach. Our new architecture  inspired by work in dependency parsing (Dozat, 2019), and which we call Rich Attention  avoids the deficien-cies of absolute and relative embeddings (Shaw et al., 2018) by avoiding embeddings entirely. Instead, we compute the order of and log distance between pairs of tokens with respect to the x and y axis on the layout grid, and adjust the pre-softmax attention scores of each pair as a direct function of these values. 3 At a high level, for each attention head at each layer (cid:96) , the model examines each pair of token representations h (cid:96)i , h (cid:96)j , whose actual order (using curly Iverson brackets) and log-distance are o ij = { i < j } and d ij = ln(1 + | i  j | ) . 3 Order on the y-axis answers the question which token is above/below the other? s ( o ) ij = o ij ln( p ij ) + (1  o ij )(1  ln( p ij )) (3) s ( d ) ij =   2 ( d ij   ij ) 2 2 (4) Finally, these are added to the usual attention score where q i = affine ( q ) ( h i ) and k j = affine ( k ) ( h j ) . The rich attention pipeline is shown in Figure 4. 5 By penalizing attention edges for violating these soft order/distance constraints, we essentially build into the model the ability to learn logical implication rules such as if x i is a noun, and x j is an adjective, and x i is related (i.e. attends ) to x j , then x j is to the left of x i . Note the unidirectionality of this rule  there could be many unrelated adjectives to the left of x i , so the converse (which this approach cannot learn) does not hold in any general sense. This is shown graphically in Figure 5. Justification. The approach taken here is not arbitrary. It can be derived algebraically from the probability mass/density functions of the distributions we assume for each feature, and the assumption that a query's attention vector represents a probability distribution. Traditional dot product attention and relative position biases (Raffel et al., 2020) can likewise be derived from this method, providing incidental justification for the approach. Consider the following, letting L ( X ) = ln( P ( X )) for brevity: 4  is a learned temperature scalar unique to each head. 5 The affine functions in Eqs. (1, 2) can optionally take the reduced-rank query/key terms q i , k j as input instead of the layer input h (cid:96)i , h (cid:96)j without sacrificing theoretical motivation. We take this approach for speed. ij | h i , h j ) = P ( h i , h j | a ij ) P ( a ij ) (cid:80) j (cid:48) [ P ( h i , h j (cid:48) | a ij (cid:48) ) P ( a ij (cid:48) )] = exp ( L ( h i , h j | a ij ) + L ( a ij )) (cid:80) j (cid:48) exp ( L ( h i , h j (cid:48) | a ij (cid:48) ) + L ( a ij (cid:48) )) = soft max j (cid:48) ( L ( h i , h j (cid:48) | a ij (cid:48) ) + L ( a ij (cid:48) )) j (5) Here a i represents a latent categorical attention variable. Eq. (5) shows that the softmax function itself can actually be derived from posterior probabilities, by simply applying Bayes' rule and then observing that x = exp(ln( x )) That is, one need not define the posterior as being the softmax of some expression, it simply is the softmax of some expression, specifically one that falls out of the assumptions one makes (explicitly or implicitly). When we plug the Gaussian probability density function into L ( h i , h j | a ij ) , the expression sim-plifies to dot-product attention (with one additional fancy bias term); we show this in Appendix C. If we assume L ( a ij ) is uniform, then it divides out of the softmax and we can ignore it. If we assume it follows a Bernoulli distribution  such that L ( a ij = 1; p ij ) = ln( p ij )  it becomes equivalent to a learned bias matrix B . 6 Now, if we assume there is another feature f ij that conditions the presence of attention, such as the order or distance of i and j , then we can use the same method to derive a parametric expression describing its impact on the attention probability. The new term can be expanded by explicating assumptions about the distributions that govern P ( f ij | h i , h j , a ij ) and simplifying the expression that results from substituting their probability functions. If f ij is binary, then this process yields Eq. (3), and if ln( f ij ) is normally distributed, we reach Eq. (4), as derived in Appendix C. Given multiple conditionally independent features  such as the order and distance  their individual scores can be calculated in this way and summed. Furthermore, relative position biases (Raffel et al., 2020) can thus be understood in this framework as binary features (e.g. f ij = { i  j =  2 } ) that are conditionally independent of h i , h j given a ij , meaning that L ( f ij | h i , h j , a ij ) = L ( f ij | a ij ) . We call this new attention paradigm Rich Attention because it allows the attention mechanism to be enriched with an arbitrary set of low-level features. We use it to add order/distance features with respect to the x and y axes of a grid  but it can also be used in a standard text transformer to encode order/distance/segment information, or it could be used in an image transformer (Parmar et al., 2018) to encode relative pixel angle/distance information 7 , without resorting to lossy quantization and finite embedding tables. The key to sparsifying attention mechanisms in ETC (Ainslie et al., 2020) for long sequence modeling is to have every token only attend to tokens that are within a pre-specified local radius in the serialized sequence. The main drawback to ETC in form understanding is that imperfect serialization sometimes results in entities being serialized too far apart from each other to attend in the local-local attention component (i.e. outside the local radius). A naive solution is to increase the local radius in ETC. However, it sacrifices the efficiency for modeling long sequences. Also, the self-attention may not be able to fully identify relevant tokens when there are many distractors (Figure 9; Serrano and Smith, 2019). To alleviate the issue, we construct a graph to connect nearby tokens in a form document. We design the edges of the graph based on strong induc-7 The von Mises or wrapped normal distribution would be most appropriate for angular features. tive biases so that they have higher probabilities of belonging to the same entity type (Figure 2(c) and 6). Then, for each token, we obtain its Super-Token embedding by applying graph convolutions along these edges to aggregate semantically meaningful information from its neighboring tokens. We use these super-tokens as input to the Rich Attention ETC for sequential tagging. This means that even though an entity may have been broken up into multiple segments due to poor serialization, the super-tokens learned by the graph convolutional network will have recovered much of the context of the entity phrase. We next introduce graph construction and the learning algorithm. Node Definition. Given a document with N tokens denoted by T = { t 1 , t 2 , . . . t N } , we let t k refer to the k -th token in a text sequence returned by the OCR engine. The OCR engine generates the bounding box sizes and locations for all tokens, as well as the text within each box. We define node input representation for all tokens T as vertices V = { v 1 , v 2 , . . . v N } , where v k concatenates attributes available for t k . In our design, we use three common input modalities: (a) one-hot word embeddings, (b) spatial embeddings from the normalized Cartesian coordinate values of the four corners and height and width of a token bounding box (Qian et al., 2019; Davis et al., 2019; Liu et al., 2019a). The benefit of representing tokens in this way is that one can add more attributes to a vertex by simple concatenation without changing the macro graph architecture. Edge Definition. While the vertices V represent tokens in a document, the edges characterize the relationship between all pairs of vertices. Precisely, we define directed edge embeddings for a set of edges E , where each edge e kl connects two vertices v k and v l , concatenating quantitative edge attributes. In our design, the edge embedding is composed of the relative distance between the centers, top left corners, and bottom right corners of the token bounding boxes. The embedding also contains the shortest distances between the bounding boxes along the horizontal and vertical axis. Finally, we include the height and width aspect ratio of v k , v l , and the bounding box that covers both of them. Graph construction. After contructing edge embeddings, we need discrete graphs to define connec-tivities. One approach would be to create k-Nearest-Neighbors graphs (Zhang et al., 2020)  but these Figure 6: An illustration of the word-level  -skeleton graph of a FUNSD document, which is a sparse but connected graph. may contain isolated components, which is not ideal for information propagation. Instead, we construct graphs using the  -skeleton algorithm (Kirk-patrick and Radke, 1985) with  = 1 , which is found useful for document understanding in Wang et al. (2022); Lee et al. (2021). It essentially creates a ball-of-sight graph with a linearly-bounded number of edges while also guaranteeing global connectivity as shown in Figure 6. More examples of constructed  -skeleton graphs can be found in Figure 11 in the Appendix. Message passing. Graph message-passing is the key to propagating representations along the edges defined by the inductive bias,  -skeleton, that are free from the left-to-right top-to-bottom form document serialization. In our design, we perform graph convolutions (GCN; Gilmer et al., 2017) on concatenated features from pairs of neighboring nodes and edges connecting them. Hence the graph embedding is directly learned from back-propagation in irregular patterns of tokens in documents. We evaluate how the two proposed structural encoding components, Rich Attention and Super-Tokens, impact the overall performance of form-like document key information extraction. We perform extensive experiments on three standard benchmarks 8 8 We note that SROIE (Huang et al., 2019) and Kleister-NDA (Gralinski et al., 2020) are designed for key-value pair 3740 Dataset Method P R F1 Image #Params Pre-training Size CORD SPADE (Hwang et al., 2021) -91.5 110M BERT-multilingual UniLMv2 (Bao et al., 2020) 91.23 92.89 92.05 355M 160GB LayoutLMv1 (Xu et al., 2021) 94.32 95.54 94.93 343M 11M DocFormer (Appalaraju et al., 2021) 96.46 96.14 96.30 502M 5M LayoutLMv2 (Xu et al., 2021) 95.65 96.37 96.01 (cid:88) 426M 11M TILT (Powalski et al., 2021) -96.33 (cid:88) 780M 1.1M DocFormer (Appalaraju et al., 2021) 97.25 96.74 96.99 (cid:88) 536M 5M FormNet (ours) 98.02 96.55 97.28 345M 0.7M (9GB) FUNSD SPADE (Hwang et al., 2021) -70.5 110M BERT-multilingual UniLMv2 (Bao et al., 2020) 67.80 73.91 70.72 355M 160GB LayoutLMv1 (Xu et al., 2020) 75.36 80.61 77.89 343M 11M DocFormer (Appalaraju et al., 2021) 81.33 85.44 83.33 502M 5M LayoutLMv1 (Xu et al., 2020) 76.77 81.95 79.27 (cid:88) 160M 11M LayoutLMv2 (Xu et al., 2021) 83.24 85.19 84.20 (cid:88) 426M 11M DocFormer (Appalaraju et al., 2021) 82.29 86.94 84.55 (cid:88) 536M 5M FormNet (ours) 85.21 84.18 84.69 217M 0.7M (9GB) Payment NeuralScoring (Majumder et al., 2020) -87.80 0 FormNet (ours) 92.70 91.69 92.19 217M 0 Table 1: Entity-level precision, recall, and F1 score comparisons on three standard benchmarks. CORD. We evaluate on CORD (Park et al., 2019), which stands for the Consolidated Receipt Dataset for post-OCR parsing. The annotations are provided in 30 fine-grained semantic entities such as store name, menu price, table number, discount, etc. We use the standard evaluation set that has 800 training, 100 validation, and 100 test samples. FUNSD. FUNSD (Jaume et al., 2019) is a public dataset for form understanding in noisy scanned documents. It is a subset of the Truth Tobacco Industry Document (TTID) 9 . The dataset consists of 199 annotated forms with 9,707 entities and 31,485 word-level annotations for 4 entity types: header, question, answer, and other. We use the official 75-25 split for the training and test sets. Payment. We use the large-scale payment data (Majumder et al., 2020) that consists of around 10K documents and 7 semantic entity labels from human annotators. The corpus comes from different vendors with different layout templates. We follow the same evaluation protocol and dataset splits used in Majumder et al. (2020). extraction instead of direct entity extraction. We leave the work of modifying FormNet for key-value pair extraction in the future. Given a document, we first use the BERT-multilingual vocabulary to tokenize the extracted OCR words. Super-tokens are then generated by direct graph embedding on these 2D tokens. Next, we use ETC transformer layers to continue to process the super-tokens based on the serialization provided by the corresponding datasets. Please see Appendix A for implementation details. Nets and scale up the FormNet family with different numbers of hidden units and attention heads to obtain FormNet-A1 (512 hidden units and 8 attention heads), A2 (768 hidden units and 12 attention heads), and A3 (1024 hidden units and 16 attention heads). Ablations on the FormNets can be found in Figure 7 and 8, and Table 4 in Appendix. MLM Pre-training. Following Appalaraju et al. (2021), we collect around 700k unlabeled form documents for unsupervised pre-training. We adopt the Masked Language Model (MLM) objective (Taylor, 1953; Devlin et al., 2019) to pre-train the networks. This forces the networks to reconstruct randomly masked tokens in a document to learn the underlying semantics of language from the pre-training corpus. We train the models from scratch using Adam optimizer with batch size of 512. The learning rate is set to 0.0002 with a warm-up proportion of 0.01. Fine-tuning. We fine-tune all models in the experiments using Adam optimizer with batch size of 8. The learning rate is set to 0.0001 without warmup. We use cross-entropy loss for the multi-class BIOES tagging tasks. The fine-tuning is conducted on Tesla V100 GPUs for approximately 10 hours on the largest corpus. Note that we only apply the MLM pre-training for the experiments on CORD and FUNSD as in Xu et al. (2020, 2021). For the experiments on Payment, we follow Majumder et al. (2020) to directly train all networks from scratch without pre-training. Benchmark Comparison. Table 1 lists the results that are based on the same evaluation proto-cal 10 . The proposed FormNet achieves the new best F1 scores on CORD, FUNSD, and Payment benchmarks. Figure 7 shows model size vs. F1 score for all recent approaches. On CORD and FUNSD, FormNet-A2 (Table 4 in Appendix) outperforms the most recent DocFormer (Appalaraju et al., 2021) while using a 2.5x smaller model and 7.1x less unlabeled pre-training documents. On the larger CORD, FormNet-A3 continues to improve the performance to the new best 97.28% F1. In addition, we observe no difficulty training the FormNet from scratch on the Payment dataset. These demonstrate the parameter efficiency and the training sample efficiency of the proposed FormNet. Effect of Structural Encoding in Pre-training. We study the importance of the proposed Rich Attention and Super-Token by GCN on the large-scale MLM pre-training task across three FormNets as summarized in Figure 8. Both Rich Attention and GCN components improve upon the ETC (Ainslie et al., 2020) baseline on reconstructing the masked tokens by a large margin, showing the effectiveness of their structural encoding capability on form documents. The best performance is obtained by incorporating both. Effect of Structural Encoding in Fine-tuning. We ablate the effect of the proposed Rich Attention and Super-Tokens by GCN on the fine-tuning tasks and measure their entity-level precision, recall, and F1 scores. In Table 2, we see that both Rich Attention and GCN improve upon the ETC (Ainslie et al., 2020) baseline on all benchmarks. In particular, Rich Attention brings 4.46 points and GCN brings 4.24 points F1 score improvement over the jumder et al., 2020). ETC baseline on CORD. We also see a total of 5.3 points increase over the baseline when using both components, showing their orthogonal effectiveness of encoding structural patterns. More ablation can be found in Section B and Table 5 in Appendix. Using BertViz (Vig, 2019), we visualize the local-to-local attention scores for specific examples of the CORD dataset for the ETC baseline and the ETC+RichAtt+GCN (FormNet) models. Qualitatively in Figure 9, we notice that the tokens attend primarily to other tokens within the same visual block for ETC+RichAtt+GCN. Moreover for that model, specific attention heads are attending to tokens aligned horizontally, which is a strong signal of meaning for form documents. No clear attention pattern emerges for the ETC model, suggesting the Rich Attention and Super-Token by GCN enable the model to learn the structural cues and leverage layout information effectively. More visualization examples are given in the Appendix E. We also show sample model outputs in Figure 10. We present a novel model architecture for key entity extraction for forms, FormNet. We show that the proposed Rich Attention and Super-Token components help the ETC transformer to excel at form understanding in spite of noisy serialization, as evidenced quantitatively by its state-of-the-art performance on three benchmarks and qualitatively by its more sensible attention patterns. In the future, we would like to explore multi-modality input such as images.
abstain abstain objective objective method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain abstain abstain result abstain other other other other other other other other other other other other other other method other method other other method method other method other other other other method objective other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method other abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain method abstain method abstain method method abstain method abstain abstain abstain abstain method method abstain method abstain abstain method abstain method abstain method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain method abstain abstain abstain abstain objective objective objective