Abstract Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895  0.8420), CORD (0.9493  0.9601), SROIE (0.9524  0.9781), Kleister-NDA (0.8340  0.8520), RVL-CDIP (0.9443  0.9564), and DocVQA (0.7295  0.8672). We made our model and code publicly available at https://aka.ms /layoutlmv2 . Visually-rich Document Understanding (VrDU) aims to analyze scanned/digital-born business documents (images of invoices, forms in PDF format, etc.) where structured information can be automatically extracted and organized for many business  Equal contributions during internship at MSRA applications. Distinct from conventional information extraction tasks, the VrDU task relies on not only textual information but also visual and layout information that is vital for visually-rich documents. Different types of documents indicate that the text fields of interest located at different positions within the document, which is often determined by the style and format of each type as well as the document content. Therefore, to accurately recognize the text fields of interest, it is inevitable to take advantage of the cross-modality nature of visually-rich documents, where the textual, visual, and layout information should be jointly modeled and learned end-to-end in a single framework. The recent progress of VrDU lies primarily in two directions. The first direction is usually built on the shallow fusion between textual and visual/layout/style information (Yang et al., 2017; Liu et al., 2019; Sarkhel and Nandi, 2019; Yu et al., 2020; Majumder et al., 2020; Wei et al., 2020; Zhang et al., 2020). These approaches leverage the pre-trained NLP and CV models individually and combine the information from multiple modalities for supervised learning. Although good performance has been achieved, the domain knowledge of one document type cannot be easily transferred into another, so that these models often need to be re-trained once the document type is changed. Thereby the local invariance in general document layout (key-value pairs in a left-right layout, tables in a grid layout, etc.) cannot be fully exploited. To this end, the second direction relies on the deep fusion among textual, visual, and layout information from a great number of unlabeled documents in different domains, where pre-training techniques play an important role in learning the cross-modality interaction in an end-to-end fashion (Lockard et al., 2020; Xu et al., 2020). In this way, the pre-trained models absorb cross-modal knowledge from different document types, where the local invariance among these layouts and styles is preserved. Furthermore, when the model needs to be transferred into another domain with different document formats, only a few labeled samples would be suf-ficient to fine-tune the generic model in order to achieve state-of-the-art accuracy. Therefore, the proposed model in this paper follows the second direction, and we explore how to further improve the pre-training strategies for the VrDU tasks. In this paper, we present an improved version of LayoutLM (Xu et al., 2020), aka LayoutLMv2 . Different from the vanilla LayoutLM model where visual embeddings are combined in the fine-tuning stage, we integrate the visual information in the pre-training stage in LayoutLMv2 by taking advantage of the Transformer architecture to learn the cross-modality interaction between visual and textual information. In addition, inspired by the 1-D relative position representations (Shaw et al., 2018; Raffel et al., 2020; Bao et al., 2020), we propose the spatial-aware self-attention mechanism for LayoutLMv2, which involves a 2-D relative position representation for token pairs. Different from the absolute 2-D position embeddings that LayoutLM uses to model the page layout, the relative position embeddings explicitly provide a broader view for the contextual spatial modeling. For the pre-training strategies, we use two new training objectives for LayoutLMv2 in addition to the masked visual-language modeling. The first is the proposed text-image alignment strategy, which aligns the text lines and the corresponding image regions. The second is the text-image matching strategy popular in previous vision-language pre-training models (Tan and Bansal, 2019; Lu et al., 2019; Su et al., 2020; Chen et al., 2020; Sun et al., 2019), where the model learns whether the document image and textual content are correlated. We select six publicly available benchmark datasets as the downstream tasks to evaluate the performance of the pre-trained LayoutLMv2 model, which are the FUNSD dataset (Jaume et al., 2019) for form understanding, the CORD dataset (Park et al., 2019) and the SROIE dataset (Huang et al., 2019) for receipt understanding, the Kleister-NDA dataset (Gralinski et al., 2020) for long document understanding with a complex layout, the RVL-CDIP dataset (Harley et al., 2015) for document image classification, and the DocVQA dataset (Mathew et al., 2021) for visual question answering on document images. Experiment results show that the LayoutLMv2 model significantly outperforms strong baselines, including the vanilla LayoutLM, and achieves new state-of-the-art results in all of these tasks. We propose a multi-modal Transformer model to integrate the document text, layout, and visual information in the pre-training stage, which learns the cross-modal interaction end-to-end in a single framework. Meanwhile, a spatial-aware self-attention mechanism is integrated into the Transformer architecture. In addition to the masked visual-language model, we add text-image alignment and text-image matching as the new pre-training strategies to enforce the alignment among different modalities. LayoutLMv2 significantly outperforms and achieves new SOTA results not only on the conventional VrDU tasks but also on the VQA task for document images, which demonstrates the great potential for the multi-modal pre-training for VrDU. In this section, we will introduce the model architecture and the multi-modal pre-training tasks of LayoutLMv2, which is illustrated in Figure 1. We build a multi-modal Transformer architecture as the backbone of LayoutLMv2, which takes text, visual, and layout information as input to establish deep cross-modal interactions. We also introduce a spatial-aware self-attention mechanism to the model architecture for better modeling the document layout. Detailed descriptions of the model are as follows. Text Embedding Following the common practice, we use WordPiece (Wu et al., 2016) to tokenize the OCR text sequence and assign each token to a certain segment s i  { [A] , [B] } . Then, we add [CLS] at the beginning of the sequence and [SEP] at the end of each text segment. Extra [PAD] tokens are appended to the end so that the Transformer Layers with Spatial-Aware Self-Attention Mechanism #3 Text-Image Matching #2 Text-Image Alignment Pre-trainingObjectives Visual/Text Token Representations [SEP] V1 V2 V3 V4 [CLS] Matched T1 Covered T3 Covered T5 NotCovered T6 NotCovered T7 NotCovered T4 T2 #1 Masked Visual-Language Modeling 1D Position Embeddings 8 0 1 2 3 0 1 3 5 6 7 4 2 V2 V1 V3 V4 Document Page with Covered OCR Lines Document Page OCR/PDF Parser Visual Encoder V1 V2 V3 V4 Feature Maps OCR Lines Line 1 (covered): Line 2 (not covered): T1 [MASK] T3 [MASK] T5 T6 T7 Visual/Text Token Embeddings 2D Position Embeddings [SEP] Box PAD V1 Box V1 V2 Box V2 V3 Box V3 V4 Box V4 [CLS] Box PAD T1 Box T1 T3 Box T3 T5 Box T5 T6 Box T6 T7 Box T7 Box T4 Box T2 Line Covering TokenMasking [MASK] [MASK] [MASK] [MASK] Segment Embeddings A C C C C A A A A A A A A Figure 1: An illustration of the model architecture and pre-training strategies for LayoutLMv2 final sequence's length is exactly the maximum sequence length L . The final text embedding is the sum of three embeddings. Token embedding represents the token itself, 1D positional embedding represents the token index, and segment embedding is used to distinguish different text segments. Formally, we have the i -th ( 0  i < L ) text embedding t i = TokEmb( w i )+PosEmb1D( i )+SegEmb( s i ) Visual Embedding Although all information we need is contained in the page image, the model has difficulty capturing detailed features in a single information-rich representation of the entire page. Therefore, we leverage the output feature map of a CNN-based visual encoder, which converts the page image to a fixed-length sequence. We use ResNeXt-FPN (Xie et al., 2017; Lin et al., 2017) architecture as the backbone of the visual encoder, whose parameters can be updated through backpropagation. Given a document page image I , it is resized to 224  224 then fed into the visual backbone. After that, the output feature map is average-pooled to a fixed size with the width being W and height being H . Next, it is flattened into a visual embedding sequence of length W  H . The sequence is named VisTokEmb( I ) . A linear projection layer is then applied to each visual token embedding to unify the dimensionality with the text embeddings. Since the CNN-based visual backbone cannot capture the positional information, we also add a 1D positional embedding to these visual token embeddings. The 1D positional embedding is shared with the text embedding layer. For the segment embedding, we attach all visual tokens to the visual segment [C] . The i -th ( 0  i < W H ) visual embedding can be represented as v i = Proj (cid:0) VisTokEmb( I ) i (cid:1) +PosEmb1D( i )+SegEmb( [C] ) Layout Embedding The layout embedding layer is for embedding the spatial layout information represented by axis-aligned token bounding boxes from the OCR results, in which box width and height together with corner coordinates are iden-tified. Following the vanilla LayoutLM, we normalize and discretize all coordinates to integers in the range [0 , 1000] , and use two embedding layers to embed x -axis features and y -axis features separately. Given the normalized bounding box of the i -th ( 0  i < W H + L ) text/visual token box i = ( x min , x max , y min , y max , width, height ) , the layout embedding layer concatenates six bounding box features to construct a token-level 2D positional embedding, aka the layout embedding l i = Concat (cid:0) PosEmb2D x ( x min , x max , width ) , PosEmb2D y ( y min , y max , height ) (cid:1) Note that CNNs perform local transformation, thus the visual token embeddings can be mapped back to image regions one by one with neither overlap nor omission. When calculating bounding boxes, the visual tokens can be treated as evenly divided grids. An empty bounding box box PAD = (0 , 0 , 0 , 0 , 0 , 0) is attached to special tokens [CLS] , [SEP] and [PAD] . Multi-modal Encoder with Spatial-Aware Self-Attention Mechanism The encoder concatenates visual embeddings { v 0 , ..., v WH  1 } and text embeddings { t 0 , ..., t L  1 } to a unified sequence and fuses spatial information by adding the layout embeddings to get the i -th ( 0  i < W H + L ) first layer input x (0) i = X i + l i , where X = { v 0 , ..., v WH  1 , t 0 , ..., t L  1 } Following the architecture of Transformer, we build our multi-modal encoder with a stack of multi-head self-attention layers followed by a feed-forward network. However, the original self-attention mechanism can only implicitly capture the relationship between the input tokens with the absolute position hints. In order to efficiently model local invariance in the document layout, it is necessary to insert relative position information explicitly. Therefore, we introduce the spatial-aware self-attention mechanism into the self-attention layers. For simplicity, the following description is for a single head in a single self-attention layer with hidden size of d head and projection matrics WQ , WK , WV . The original self-attention mechanism captures the correlation between query x i and key x j by projecting the two vectors and calculating the attention score  ij = 1  d head (cid:0) x i WQ (cid:1) (cid:0) x j WK (cid:1) T Considering the large range of positions, we model the semantic relative position and spatial relative position as bias terms to prevent adding too many parameters. Similar practice has been shown effective on text-only Transformer architectures (Raffel et al., 2020; Bao et al., 2020). Let b (1D) , b (2D x ) and b (2D y ) denote the learnable 1D and 2D relative position biases respectively. The biases are different among attention heads but shared in all encoder layers. Assuming ( x i , y i ) anchors the top left corner coordinates of the i -th bounding box, we obtain the spatial-aware attention score  (cid:48) ij =  ij + b (1D) j  i + b (2D x ) x j  x i + b (2D y ) y j  y i Finally, the output vectors are represented as the weighted average of all the projected value vectors with respect to normalized spatial-aware attention scores h i = (cid:88) j exp (cid:16)  (cid:48) ij (cid:17) (cid:80) k exp (cid:0)  (cid:48) ik (cid:1) x j WV 2.2 Pre-training Tasks Masked Visual-Language Modeling Similar to the vanilla LayoutLM, we use the Masked Visual-Language Modeling (MVLM) to make the model learn better in the language side with the cross-modality clues. We randomly mask some text tokens and ask the model to recover the masked tokens. Meanwhile, the layout information remains unchanged, which means the model knows each masked token's location on the page. The output representations of masked tokens from the encoder are fed into a classifier over the whole vocabulary, driven by a cross-entropy loss. To avoid visual clue leakage, we mask image regions corresponding to masked tokens on the raw page image input before feeding it into the visual encoder. Text-Image Alignment To help the model learn the spatial location correspondence between image and coordinates of bounding boxes, we propose the Text-Image Alignment (TIA) as a fine-grained cross-modality alignment task. In the TIA task, some tokens lines are randomly selected, and their image regions are covered on the document image. We call this operation covering to avoid confusion with the masking operation in MVLM. During pretraining, a classification layer is built above the encoder outputs. This layer predicts a label for each text token depending on whether it is covered, i.e., [Covered] or [Not Covered] , and computes the binary cross-entropy loss. Considering the input image's resolution is limited, and some document elements like signs and bars in a fig-ure may look like covered text regions, the task of finding a word-sized covered image region can be noisy. Thus, the covering operation is performed at the line-level. When MVLM and TIA are performed simultaneously, TIA losses of the tokens masked in MVLM are not taken into account. This prevents the model from learning the useless but straightforward correspondence from [MASK] to [Covered] . Text-Image Matching Furthermore, a coarse-grained cross-modality alignment task, Text-Image Matching (TIM) is applied to help the model learn the correspondence between document image and textual content. We feed the output representation at [CLS] into a classifier to predict whether the image and text are from the same document page. Regular inputs are positive samples. To construct a negative sample, an image is either replaced by a page image from another document or dropped. To prevent the model from cheating by finding task features, we perform the same masking and covering operations to images in negative samples. The TIA target labels are all set to [Covered] in negative samples. We apply the binary cross-entropy loss in the optimization process. In order to pre-train and evaluate LayoutLMv2 models, we select datasets in a wide range from the visually-rich document understanding area. Following LayoutLM, we use IIT-CDIP Test Collection (Lewis et al., 2006) as the pre-training dataset. Six datasets are used as down-stream tasks. The FUNSD (Jaume et al., 2019), CORD (Park et al., 2019), SROIE (Huang et al., 2019) and Kleister-NDA (Gralinski et al., 2020) datasets define entity extraction tasks that aim to extract the value of a set of pre-defined keys, which we formalize as a sequential labeling task. RVL-CDIP (Harley et al., 2015) is for document image classification. DocVQA (Mathew et al., 2021), as the name suggests, is a dataset for visual question answering on document images. Statistics of datasets are shown in Table 1. Refer to the Appendix for details. Following the typical pre-training and fine-tuning strategy, we update all parameters including the visual encoder layers, and train whole models end-to-end for all the settings. Training details can be found in the Appendix. Pre-training LayoutLMv2 We train LayoutLMv2 models with two different parameter sizes. We use a 12-layer 12-head Transformer encoder and set hidden size d = 768 in LayoutLMv2 BASE . While in the LayoutLMv2 LARGE , the encoder has 24 Transformer layers with 16 heads and d = 1024 . Visual backbones in the two models are based on the same ResNeXt101-FPN architecture. The numbers of parameters are 200M and 426M approximately for LayoutLMv2 BASE and LayoutLMv2 LARGE , respectively. For the encoder along with the text embedding layer, LayoutLMv2 uses the same architecture as UniLMv2 (Bao et al., 2020), thus it is initialized from UniLMv2. For the ResNeXt-FPN part in the visual embedding layer, the backbone of a Mask-RCNN (He et al., 2017) model trained on Pub-LayNet (Zhong et al., 2019) is leveraged. 1 The rest of the parameters in the model are randomly initialized. During pre-training, we sample pages from the IIT-CDIP dataset and select a random sliding window of the text sequence if the sample is too long. We set the maximum sequence length L = 512 and assign all text tokens to the segment [A] . The output shape of the average pooling layer is set to W = H = 7 , so that it transforms the feature map into 49 visual tokens. In MVLM, 15% text tokens are masked among which 80% are replaced by a special token [MASK] , 10% are replaced by a random token sampled from the whole vocabulary, and 1 MaskRCNN ResNeXt101 32x8d FPN 3X setting in https://github.com/hpanwar08/detectron2 Model FUNSD CORD SROIE Kleister-NDA BERTBASE 0.6026 0.8968 0.9099 0.7790 UniLMv2 BASE 0.6648 0.9092 0.9459 0.7950 BERTLARGE 0.6563 0.9025 0.9200 0.7910 UniLMv2 LARGE 0.7072 0.9205 0.9488 0.8180 LayoutLM BASE 0.7866 0.9472 0.9438 0.8270 LayoutLM LARGE 0.7895 0.9493 0.9524 0.8340 LayoutLMv2 BASE 0.8276 0.9495 0.9625 0.8330 LayoutLMv2 LARGE 0.8420 0.9601 0.9781 0.8520 BROS (Hong et al., 2021) 0.8121 0.9536 0.9548  SPADE (Hwang et al., 2020)  0.9150   PICK (Yu et al., 2020)   0.9612  TRIE (Zhang et al., 2020)   0.9618  Top-1 on SROIE Leaderboard (until 2020-12-24)   0.9767  RoBERTa BASE in (Gralinski et al., 2020)    0.7930 Table 2: Entity-level F1 scores of the four entity extraction tasks: FUNSD, CORD, SROIE and Kleister-NDA. 10% remains the same. In TIA, 15% of the lines are covered. In TIM, 15% images are replaced, and 5% are dropped. Fine-tuning LayoutLMv2 We use the [CLS] output along with pooled visual token representations as global features in the document-level classification task RVL-CDIP. For the extractive question answering task DocVQA and the other four entity extraction tasks, we follow common practice like (Devlin et al., 2019) and build task specified head layers over the text part of LayoutLMv2 outputs. In the DocVQA paper, experiment results show that the BERT model fine-tuned on the SQuAD dataset (Rajpurkar et al., 2016) outperforms the original BERT model. Inspired by this fact, we add an extra setting, which is that we first fine-tune LayoutLMv2 on a question generation (QG) dataset followed by the DocVQA dataset. The QG dataset contains almost one million question-answer pairs generated by a generation model trained on the SQuAD dataset. Baselines We select three baseline models in the experiments to compare LayoutLMv2 with the text-only pre-trained models as well as the vanilla LayoutLM model. Specifically, we compare LayoutLMv2 with BERT (Devlin et al., 2019), UniLMv2 (Bao et al., 2020), and LayoutLM (Xu et al., 2020) for all the experiment settings. We use the publicly available PyTorch models for BERT (Wolf et al., 2020) and LayoutLM, and use our in-house implementation for the UniLMv2 models. For each baseline approach, experiments are conducted using both the BASE and LARGE parameter settings. Entity Extraction Tasks Table 2 shows the model accuracy on the four datasets FUNSD, CORD, SROIE, and Kleister-NDA, which we regard as sequential labeling tasks evaluated using entity-level F1 score. We report the evaluation results of Kleister-NDA on the validation set because the ground-truth labels and the submission website for the test set are not available right now. For text-only models, the UniLMv2 models outperform the BERT models by a large margin in terms of the BASE and LARGE settings. For text+layout models, the LayoutLM family, especially the LayoutLMv2 models, brings significant performance improvement over the text-only baselines. Compared to the baselines, the LayoutLMv2 models are superior to the SPADE (Hwang et al., 2020) decoder method, as well as the text+layout pre-training approach BROS (Hong et al., 2021) that is built on the SPADE decoder, which demonstrates the effectiveness of our modeling approach. Moreover, with the same modal information, our LayoutLMv2 models also outperform existing multi-modal approaches PICK (Yu et al., 2020), TRIE (Zhang et al., 2020) and the previous top-1 method on the leaderboard, 2 confirming the effectiveness of our pre-training for text, layout, and visual information. The best performance on all the four datasets is achieved by 2 Unpublished results, the leaderboard is available at https://rrc.cvc.uab.es/?ch=13&com=eval uation&task=3 Model Accuracy BERTBASE 89.81% UniLMv2 BASE 90.06% BERTLARGE 89.92% UniLMv2 LARGE 90.20% LayoutLM BASE (w/ image) 94.42% LayoutLM LARGE (w/ image) 94.43% LayoutLMv2 BASE 95.25% LayoutLMv2 LARGE 95.64% VGG-16 (Afzal et al., 2017) 90.97% Single model (Das et al., 2018) 91.11% Ensemble (Das et al., 2018) 92.21% InceptionResNetV2 (Szegedy et al., 2017) 92.63% LadderNet (Sarkhel and Nandi, 2019) 92.77% Single model (Dauphinee et al., 2019) 93.03% Ensemble (Dauphinee et al., 2019) 93.07% Table 3: Classification accuracy on the RVL-CDIP dataset the LayoutLMv2 LARGE , which illustrates that the multi-modal pre-training in LayoutLMv2 learns better from the interactions from different modalities, thereby leading to the new SOTA on various document understanding tasks. RVL-CDIP Table 3 shows the classification accuracy on the RVL-CDIP dataset, including text-only pre-trained models, the LayoutLM family as well as several image-based baseline models. As shown in the table, both the text and visual information are important to the document image classification task because document images are text-intensive and represented by a variety of layouts and formats. Therefore, we observed that the LayoutLM family outperforms those text-only or image-only models as it leverages the multi-modal information within the documents. Specifically, the LayoutLMv2 LARGE model significantly improves the classification accuracy by more than 1.2% point over the previous SOTA results, which achieves an accuracy of 95.64%. This also verifies that the pre-trained LayoutLMv2 model benefits not only the information extraction tasks in document understanding but also the document image classification task through effective multi-model training. DocVQA Table 4 lists the Average Normalized Levenshtein Similarity (ANLS) scores on the DocVQA dataset of text-only baselines, LayoutLM family models, and the previous top-1 on the leaderboard. With multi-modal pre-training, LayoutLMv2 models outperform LayoutLM models and text-only baselines by a large margin when fine-Model Fine-tuning set ANLSBERTBASE train 0.6354 UniLMv2 BASE train 0.7134 BERTLARGE train 0.6768 UniLMv2 LARGE train 0.7709 LayoutLM BASE train 0.6979 LayoutLM LARGE train 0.7259 LayoutLMv2 BASE train 0.7808 LayoutLMv2 LARGE train 0.8348 LayoutLMv2 LARGE train + dev 0.8529 LayoutLMv2 LARGE + QG train + dev 0.8672 Top-1 (30 models ensemble) on DocVQA Leaderboard (until 2020-12-24) -0.8506 Table 4: ANLS score on the DocVQA dataset, QG denotes the data augmentation with the question generation dataset. tuned on the train set. By using all data (train + dev) as the fine-tuning dataset, the LayoutLMv2 LARGE single model outperforms the previous top-1 on the leaderboard which ensembles 30 models. 3 Under the setting of fine-tuning LayoutLMv2 LARGE on a question generation dataset (QG) and the DocVQA dataset successively, the single model performance increases by more than 1.6% ANLS and achieves the new SOTA. To fully understand the underlying impact of different components, we conduct an ablation study to explore the effect of visual information, the pretraining tasks, spatial-aware self-attention mechanism, as well as different text-side initialization models. Table 5 shows model performance on the DocVQA validation set. Under all the settings, we pre-train the models using all IIT-CDIP data for one epoch. The hyper-parameters are the same as those used to pre-train LayoutLMv2 BASE in Section 3.2. LayoutLM denotes the vanilla LayoutLM architecture in (Xu et al., 2020), which can be regarded as a LayoutLMv2 architecture without visual module and spatial-aware self-attention mechanism. X101-FPN denotes the ResNeXt101-FPN visual backbone described in Section 3.2. We first evaluate the effect of introducing visual information. From #1 to #2a, we add the visual module without changing the pre-training strategy, where results show that LayoutLMv2 pre-3 Unpublished results, the leaderboard is available at https://rrc.cvc.uab.es/?ch=17&com=eval uation&task=1 # Model Architecture Initialization SASAM MVLM TIA TIM ANLS 1 LayoutLM BASEBERTBASE (cid:88) 0.6841 2a LayoutLMv2 BASEBERTBASE + X101-FPN (cid:88) 0.6915 2b LayoutLMv2 BASEBERTBASE + X101-FPN (cid:88) (cid:88) 0.7061 2c LayoutLMv2 BASEBERTBASE + X101-FPN (cid:88) (cid:88) 0.6955 2d LayoutLMv2 BASEBERTBASE + X101-FPN (cid:88) (cid:88) (cid:88) 0.7124 3 LayoutLMv2 BASEBERTBASE + X101-FPN (cid:88) (cid:88) (cid:88) (cid:88) 0.7217 4 LayoutLMv2 BASE UniLMv2 BASE + X101-FPN (cid:88) (cid:88) (cid:88) (cid:88) 0.7421 Table 5: Ablation study on the DocVQA dataset, where ANLS scores on the validation set are reported. trained with only MVLM can leverage visual information effectively. Then, we compare the two cross-modality alignment pre-training tasks TIA and TIM. According to the four results in #2, both tasks improve the model performance substantially, and the proposed TIA benefits the model more than the commonly used TIM. Using both tasks together is more effective than using either one alone. According to this observation, we keep all the three pre-training tasks and introduce the spatial-aware self-attention mechanism (SASAM) to the model architecture. Compare the results #2d and #3, the proposed SASAM can further improve the model accuracy. Finally, in settings #3 and #4, we change the text-side initialization checkpoint from BERT to UniLMv2, and confirm that LayoutLMv2 benefits from the better initialization. In recent years, pre-training techniques have become popular in both NLP and CV areas, and have also been leveraged in the VrDU tasks. Devlin et al. (2019) introduced a new language representation model called BERT, which is designed to pre-train deep bidirectional representations from the unlabeled text by jointly conditioning on both left and right context in all layers. Bao et al. (2020) propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model. Our multi-modal Transformer architecture and the MVLM pre-training strategy extend Transformer and MLM used in these work to leverage visual information. Lu et al. (2019) proposed ViLBERT for learning task-agnostic joint representations of image content and natural language by extending the popular BERT architecture to a multi-modal two-stream model. Su et al. (2020) proposed VL-BERT that adopts the Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. Different from these vision-language pre-training approaches, the visual part of LayoutLMv2 directly uses the feature map instead of pooled ROI features, and benefits from the new TIA pre-training task. Xu et al. (2020) proposed LayoutLM to jointly model interactions between text and layout information across scanned document images, benefit-ing a great number of real-world document image understanding tasks such as information extraction from scanned documents. This work is a natural extension of the vanilla LayoutLM, which takes advantage of textual, layout, and visual information in a single multi-modal pre-training framework. In this paper, we present a multi-modal pre-training approach for visually-rich document understanding tasks, aka LayoutLMv2. Distinct from existing methods for VrDU, the LayoutLMv2 model not only considers the text and layout information but also integrates the image information in the pretraining stage with a single multi-modal framework. Meanwhile, the spatial-aware self-attention mechanism is integrated into the Transformer architecture to capture the relative relationship among different bounding boxes. Furthermore, new pre-training objectives are also leveraged to enforce the learning of cross-modal interaction among different modalities. Experiment results on 6 different VrDU tasks have illustrated that the pre-trained LayoutLMv2 model has substantially outperformed the SOTA baselines in the document intelligence area, which greatly benefits a number of real-world document understanding tasks. For future research, we will further explore the network architecture as well as the pre-training strategies for the LayoutLM family. Meanwhile, we will also investigate the language expansion to make the multi-lingual LayoutLMv2 model available for different languages, especially the non-English areas around the world. This work was supported by the National Key R&D Program of China via grant 2020AAA0106501 and the National Natural Science Foundation of China (NSFC) via grant 61976072 and 61772153.
abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result method abstain abstain objective abstain abstain method abstain objective objective objective objective method method method abstain method method other abstain abstain method method method abstain abstain abstain abstain abstain method abstain method other method other abstain abstain method abstain abstain method abstain method abstain abstain abstain method method abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain other abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain other other other objective other other other other method method abstain abstain abstain abstain objective objective other