Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective L anguagei ndependent L ayout T ransformer ( LiLT ) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure. Code and model are publicly available at https://github.com/jpWang/LiLT. Structured document understanding (SDU) aims at reading and analyzing the textual and structured information contained in scanned/digital-born documents. With the acceleration of the digitization process, it has been regarded as a crucial part of intelligent document processing and required by many real-world applications in various industries such as finance, medical treatment and insurance. Recently, inspired by the rapid development of pre-trained language models of plain texts (Devlin et al., 2019; Liu et al., 2019b; Bao et al., 2020; Chi et al., 2021), many researches on structured document pre-training (Xu et al., 2020, 2021a,b; Li et al., 2021a,b,c; Appalaraju et al., 2021) have also  Corresponding author. pushed the limit of a variety of SDU tasks. However, almost all of them only focus on pre-training and fine-tuning on the documents in a single language, typically English. This is extremely limited for other languages, especially in the case of lacking pre-training structured document data. In this regard, we consider how to make the SDU tasks enjoy language-independent benefit from the pre-training of document layout structure. Here, we give an observation as shown in Figure 1. When the layout structure remains unchanged, the substitution of language does not make obvious unnaturalness. It fully motivates us to decouple and reuse the layout invariance among different languages. Based on this inspiration, in this paper, we propose a simple yet effective L anguagei ndependent L ayout T ransformer ( LiLT ) for structured document understanding. In our framework, the text and layout information are first decoupled and joint-optimized during pre-training, and then re-coupled for fine-tuning. To ensure that the two modalities have sufficient language-independent interaction, we further propose a novel bi-directional attention complementation mechanism (BiACM) to enhance the cross-modality cooperation. Moreover, we present the key point location (KPL) and cross-modal alignment identification (CAI) tasks, which are combined with the widely-used masked visual-7747 language modeling (MVLM) to serve as our pretraining objectives. During fine-tuning, the layout flow (LiLT) can be separated and combined with the off-the-shelf pre-trained textual models (such as RoBERTa (Liu et al., 2019b), XLM-R (Conneau et al., 2020), InfoXLM (Chi et al., 2021), etc) to deal with the downstream tasks. In this way, our method decouples and learns the layout knowledge from the monolingual structured documents before generalizing it to the multilingual ones. To the best of our knowledge, the only preexisting multilingual SDU model is LayoutXLM (Xu et al., 2021b). It scraps multilingual PDF documents of 53 languages from a web crawler and introduces extra pre-processing steps to clean the collected data, filter the low-quality documents, and classify them into different languages. After this, it utilizes a heuristic distribution to sample 22 million multilingual documents, which are further combined with the 8 million sampled English ones from the IIT-CDIP (Lewis et al., 2006) dataset (11 million English documents), resulting 30 million for pre-training with the LayoutLMv2 (Xu et al., 2021a) framework. However, this process is time-consuming and laborious. On the contrary, LiLT can be pre-trained with only IIT-CDIP and then adapted to other languages. In this respect, LiLT is the first language-independent method for structured document understanding. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which substantially benefits numerous real-world SDU applications. Our main contributions can be summarized as follows:  We introduce a simple yet effective language-independent layout Transformer called LiLT for monolingual/multilingual structured document understanding. We propose BiACM to provide language-independent cross-modality interaction, along with an effective asynchronous optimization strategy for textual and non-textual flows in pre-training. Moreover, we present two new pre-training objectives, namely KPL and CAI. LiLT achieves competitive or even superior performance on various widely-used downstream benchmarks of different languages under different settings, which fully demonstrates its effectiveness. Figure 2 shows the overall illustration of our method. Given an input document image, we first use off-the-shelf OCR engines to get text bounding boxes and contents. Then, the text and layout information are separately embedded and fed into the corresponding Transformer-based architecture to obtain enhanced features. Bi-directional attention complementation mechanism (BiACM) is introduced to accomplish the cross-modality interaction of text and layout clues. Finally, the encoded text and layout features are concatenated and additional heads are added upon them, for the self-supervised pre-training or the downstream fine-tuning. The whole framework can be regarded as a parallel dual-stream Transformer. The layout flow shares a similar structure as text flow, except for the reduced hidden size and intermediate size to achieve computational efficiency. Following the common practice (Devlin et al., 2019; Xu et al., 2020), in the text flow, all text strings in the OCR results are first tokenized and concatenated as a sequence S t by sorting the corresponding text bounding boxes from the top-left to bottom-right. Intuitively, the special tokens [CLS] and [SEP] are also added at the beginning and end of the sequence respectively. After this, S t will be truncated or padded with extra [PAD] tokens until its length equals the maximum sequence length N . Finally, we sum the token embedding E token of S t and the 1D positional embedding P 1D to obtain the text embedding ET  RN  d T as: ET = LN( E token + P 1D ) , (1) where d T is the number of text feature dimension and LN is the layer normalization (Ba et al., 2016). As for the layout flow, we construct a 2D position sequence S l with the same length as the token sequence S t using the corresponding text bounding boxes. To be specific, we normalize and dis-cretize all box coordinates to integers in the range [0 , 1000] , and use four embedding layers to generate x -axis, y -axis, height, and width features separately. Given the normalized bounding boxes B = ( x min , x max , y min , y max , width, height ) , the 2D 7748 MatMul MatMul MaskOut SoftMax QTKTVT Transformer Layer i MatMul MatMul MaskOut SoftMax QLKLVL Transformer Layer i Token Embedding 1D Position Embedding + + + + + + + + t 1 t M t 2 t 4 t R t 7 t 6 t 8 1 3 2 4 5 7 6 8 2D Position Embedding 1D Position Embedding + + + + + + + + b 1 b 3 b R b 4 b 5 b 7 b 6 b M 1 3 2 4 5 7 6 8 + +  Scale The Text Flow (RoBERTa/XLM-R/InfoXLM/...) The Layout Flow (LiLT) || Pre-training Objectives MaskedVisual-Language Modeling t 3 -t 5 t 7 --r 2 r 4 -r 8 Key Point Location Cross-modalAlignment Identification (0:Mis-aligned, 1:Aligned) -0 1 0 1 -1 3 2 4 5 7 6 8 Fine-tuning Tasks Semantic Entity Recognition (H:Header, Q:Question, A:Answer, O:Other) O O H Q A Q O A -------------0 ---0 1 -------0 0 0 --0 0 0 1 -Relation Extraction (0:None, 1:Key-Value Pair)   OCR Engines Scale N l  + ||  Concatenate Add Detach (only exists in pre-training) BiACM Figure 2: The overall illustration of our framework. positional embedding P 2D  RN  d L (where d L is the number of layout feature dimension) is constructed as follows: P 2D = Linear (CAT( E x min , E x max , E y min , E y max ,E width , E height )) . Here, the E s are embedded vectors. Linear is a linear projection layer and CAT is the channel-wise concatenation operation. The special tokens [CLS] , [SEP] and [PAD] are also attached with (0,0,0,0,0,0), (1000,1000,1000,1000,0,0) and (0,0,0,0,0,0) respectively. It is worth mentioning that, for each token, we directly utilize the bounding box of the text string it belongs to, because the fine-grained token-level information is not always included in the results of some OCR engines. Since Transformer layers are permutation-invariant, here we introduce the 1D positional embedding again. The resulting layout embedding EL  RN  d L can be formulated as: EL = LN( P 2D + P 1D ) . The text embedding ET and layout embedding EL are fed into their respective sub-models to generate high-level enhanced features. However, it will considerably ignore the cross-modal interaction process if we simply combine the text and layout features at the encoder output only. The network also needs to comprehensively analyse them at earlier stages. In view of this, we propose a new bi-directional attention complementation mechanism (BiACM) to strengthen the cross-modality interaction across the entire encoding pipeline. Experiments in Section 3.2 will further verify its effectiveness. The vanilla self-attention mechanism in Transformer layers captures the correlation between query x i and key x j by projecting the two vectors and calculating the attention score as:  ij = ( x i WQ )( x j WK )   d h . Here, the description is for a single head in a single self-attention layer with hidden size of d h and projection metrics WQ , WK for simplicity. Given  Tij and  Lij of the text and layout flows located in the same head of the same layer, BiACM shares them as common knowledge, which is formulated as: (cid:102)  Tij =  Lij +  Tij , (5) (cid:102)  Lij = (cid:40)  Lij + DETACH(  Tij ) if Pre train ,  Lij +  Tij if Fine tune . In order to maintain the ability of LiLT to cooperate with different off-the-shelf text models in fine-tuning as much as possible, we heuristically adopt the detached  Tij for (cid:102)  Lij , so that the textual stream will not be affected by the gradient of non-textual one during pre-training, and its overall consistency can be preserved. Finally, the modified attention scores are used to weight the projected value vectors for subsequent modules in both flows. We conduct three self-supervised pre-training tasks to guide the model to autonomously learn joint representations with cross-modal cooperation. The details are introduced below. This task is originally derived from (Devlin et al., 2019). MVLM randomly masks some of the input tokens and the model is asked to recover them over the whole vocabulary using the output encoded features, driven by a cross-entropy loss. Meanwhile, the non-textual information remains unchanged. MVLM improves model learning on the language side with cross-modality information. The given layout embedding can also help the model better capture both interand intra-sentence relationships. We mask 15% text tokens, among which 80% are replaced by the special token [MASK] , 10% are replaced by random tokens sampled from the whole vocabulary, and 10% remain the same. We propose this task to make the model better understand layout information in the structured documents. KPL equally divides the entire layout into several regions (we set 7  7=49 regions by default) and randomly masks some of the input bounding boxes. The model is required to predict which regions the key points (top-left corner, bottom-right corner, and center point) of each box belong to using separate heads. To deal with it, the model is required to fully understand the text content and know where to put a specific word/sentence when the surrounding ones are given. We mask 15% boxes, among which 80% are replaced by (0,0,0,0,0,0), 10% are replaced by random boxes sampled from the same batch, and 10% remain the same. Cross-entropy loss is adopted. Since there may exist detection errors in the output of OCR engines, we let the model predict the discretized regions (as mentioned above) instead of the exact location. This strategy can moderately relax the punishment criterion while improving the model performance. We collect those encoded features of token-box pairs that are masked and further replaced (mis-aligned) or kept unchanged (aligned) by MVLM and KPL, and build an additional head upon them to identify whether each pair is aligned. To achieve this, the model is required to learn the cross-modal perception capacity. CAI is a binary classification task, and a cross-entropy loss is applied for it. Utilizing a unified learning rate for all model parameters to perform the end-to-end training process is the most common optimization strategy. While in our case, it will cause the layout flow to continuously update in the direction of coupling with the evolving text flow in the pre-training stage, which is harmful to the ability of LiLT to cooperate with different off-the-shelf textual models during fine-tuning. Based on this consideration, we explore multiple ratios to greatly slow down the pre-training optimization of the text stream. We also find that an appropriate reduction ratio is better than parameter freezing. Note that, we adopt a unified learning rate for end-to-end optimization during fine-tuning. The DETACH operation of BiACM is also canceled at this time, as shown in Equation 6. 3 Experiments 3.1 Pre-training Setting We pre-train LiLT on the IIT-CDIP Test Collection 1.0 (Lewis et al., 2006), which is a large-scale scanned document image dataset and contains more than 6 million documents with more than 11 million scanned document images. We use TextIn API 1 to obtain the text bounding boxes and strings for this dataset. In this paper, we initialize the text flow from the existing pre-trained English RoBERTa BASE (Liu et al., 2019b) for our document pre-training, and combine LiLT BASE with the pre-trained InfoXLM BASE (Chi et al., 2021)/a new pre-trained RoBERTa BASE for multilingual/monolingual fine-tuning. They have an equal number of self-attention layers, attention heads and maximum sequence length, which ensures that BiACM can work normally. In this BASE setting, LiLT has a 12-layer encoder with 192 hidden size, 768 feed-forward filter size and 12 attention heads, resulting 1 https://www.textin.com 7750 # Inter-modal Operation Average F1 1 CAT 0.6751 2 CAT + Co-Attention (Lu et al., 2019) 0.6276 3 CAT + BiACM 0.7963 4 CAT + BiACM  DETACH in pre-training 0.7682 5 CAT + BiACM + DETACH in fine-tuning 0.7822 6 The text flow alone (InfoXLM BASE , as shown in Table 6) 0.7207 LiLT BASE is pre-trained using Adam optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2018), with the learning rate 2  10  5 , weight decay 1  10  2 , and (  1 ,  2 ) = (0.9, 0.999). The learning rate is linearly warmed up over the first 10% steps and then linearly decayed. We set the batch size as 96 and train LiLT BASE for 5 epochs on the IIT-CDIP dataset using 4 NVIDIA A40 48GB GPUs. Considering that the complete pre-training takes a relatively long time, we pre-train LiLT BASE with 2M documents randomly sampled from IIT-CDIP for 5 epochs to conduct ablation experiments, as shown in Table 1. We first evaluate the effect of introducing BiACM. In setting (a)#1, the text and layout features are concatenated at the model output without any further interaction. Compared with (a)#6, Model Precision Recall F1 BERT BASE1 0.5469 0.6710 0.6026 RoBERTa BASE2 0.6349 0.6975 0.6648 UniLMv2 BASE3 0.6349 0.6975 0.6648 LayoutLM BASE4 0.7597 0.8155 0.7866 BROS BASE5 0.8056 0.8188 0.8121 SelfDoc 6 -0.8336 LayoutLMv2 BASE7 0.8029 0.8539 0.8276 StrucTexT BASE8 0.8568 0.8097 0.8309 DocFormer BASE9 0.8076 0.8609 0.8334  LayoutXLM BASE10 0.7913 0.8158 0.8034 LiLT[ EN-R 2 ] BASE 0.8721 0.8965 0.8841  LiLT[ InfoXLM 11 ] BASE 0.8467 0.8709 0.8586 Table 2: Comparison on the semantic entity recognition (SER) task of FUNSD (Jaume et al., 2019) dataset. we find that such a plain design results in a much worse performance than using the text flow alone. From (a)#1 to (a)#3, the significant improvement demonstrates that it is the novel BiACM that makes the transfer from monolingual to multilingual successful. Beside this, we have also tried to replace BiACM with the co-attention mechanism (Lu et al., 2019) which is widely adopted in dual-stream Transformer architecture. It can be seen as a deeper cross-modal interaction, since the keys and values from each modality are passed as input to the other modality's dot-product attention calculation. However, severe drops are observed as shown in (a)#2 vs (a)#1#3. We attribute it to the damage of such a deeper interaction to the overall consistency of the text flow in the pre-training optimization. In contrast, BiACM can maintain LiLT's cross-model cooperation ability on the basis of providing cross-modal information. Moreover, the necessity of DETACH in pre-training is proved in (a)#4 vs (a)#3. Compared (a)#3 to (a)#5, we can also infer that removing DETACH in fine-tuning leads to a better performance. Then, we compare the proposed KPL and CAI tasks. As shown in Table 1(b), both tasks improve the model performance substantially, and the proposed CAI benefits the model more than KPL. Using both tasks together is more effective than using either one alone. Finally, we explore the most suitable slow-down ratio for the pre-training optimization of the text flow. A ratio equal to 1 in (c)#1 means there is no slow-down and a unified learning rate is adopted. It can be found that the F1 scores keep rising with the growth of slow-down ratios and begin to fall when the ratio is greater than 1000. Consequently, we set the slow-down ratio as 1000 by default. To demonstrate the performance of LiLT, we conduct experiments on several widely-used monolingual datasets and the multilingual XFUND benchmark (Xu et al., 2021b). In addition to the experiments involving typical language-specific fine-tuning, we also follow the two settings designed Model Accuracy VGG-16 1 90.97% Stacked CNN Single 2 91.11% Stacked CNN Ensemble 2 92.21% InceptionResNetV2 3 92.63% LadderNet 4 92.77% Multimodal Single 5 93.03% Multimodal Ensemble 5 93.07% BERTBASE 89.81% UniLMv2 BASE 90.06% LayoutLM BASE (w/ image) 94.42% BROSBASE 95.58% SelfDoc 93.81% TILTBASE 93.50% LayoutLMv2 BASE 95.25% DocFormer BASE 96.17%  LayoutXLM BASE 95.21% LiLT[ EN-R ] BASE 95.68%  LiLT[ InfoXLM ] BASE 95.62% Table 5: Comparison on the document classification (DC) task of RVL-CDIP (Harley et al., 2015) dataset. in (Xu et al., 2021b) to demonstrate the ability to transfer knowledge among different languages, which are zero-shot transfer learning and multitask fine-tuning, for fair comparisons. Specifically, (1) language-specific fine-tuning refers to the typical fine-tuning paradigm of fine-tuning on language X and testing on language X. (2) Zero-shot transfer learning means the models are fine-tuned on English data only and then evaluated on each target language. (3) Multitask fine-tuning requires the model to fine-tune on data in all languages. We first evaluate LiLT on four widely-used monolingual datasets FUNSD (Jaume et al., 2019), CORD (Park et al., 2019), EPHOIE (Wang et al., 2021a) and RVL-CDIP (Lewis et al., 2006), and the results are shown in Table 2, 3, 4 and 5. We have found that (1) LiLT is flexible since it can work with monolingual or multilingual plain text models to deal with downstream tasks. (2) Although LiLT is designed for the transfer from monolingual to multilingual, it can surprisingly cooperate with monolingual textual models to achieve competitive or even superior performance (especially on the FUNSD dataset with only a few training samples available), compared with existing language-specific SDU models such as LayoutLMv2 and 7752 Task Model Pre-training Docs FUNSD XFUND Avg. DocFormer. (3) On these datasets which are widely adopted for monolingual evaluation, LiLT generally performs better than LayoutXLM. This fully demonstrates the effectiveness of our pre-training framework and indicates that the layout and text information can be successfully decoupled in pretraining and re-coupled in fine-tuning. Then we evaluate LiLT on language-specific fine-tuning tasks of FUNSD and the multilingual XFUND (Xu et al., 2021b), and the results are shown in Table 6. Compared with the plain text models (XLM-R/InfoXLM) or the LayoutXLM model pre-trained with 30M multilingual structured documents, LiLT achieves the highest F1 scores on both the SER and RE tasks of each language while using 11M monolingual data. This significant improvement shows LiLT's capability to transfer language-independent knowledge from pre-training to downstream tasks. The results of cross-lingual zero-shot transfer are presented in Table 7. It can be observed that the LiLT model transfers the most knowledge from English to other languages, and significantly outperforms its competitors. This fully verifies that LiLT can capture the common layout invariance among different languages. Moreover, LiLT has never seen non-English documents before evaluation under this setting, while the LayoutXLM model has been pre-trained with them. This is to say, LiLT faces a stricter cross-lingual zero-shot transfer scenario but achieves better performance. Table 8 shows the results of multitask learning. In this setting, the pre-trained LiLT model is simultaneously fine-tuned with all eight languages and evaluated for each specific language. We observe that this setting further improves the model performance compared to the language-specific fine-tuning, which confirms that SDU can benefit from commonalities in the layout of multilingual structured documents. In addition, LiLT once again outperforms its counterparts by a large margin. During the past decade, deep learning methods became the mainstream for document understanding tasks (Yang et al., 2017; Augusto Borges Oliveira et al., 2017; Siegel et al., 2018). Grid-based methods (Katti et al., 2018; Denk and Reisswig, 2019; Lin et al., 2021) were proposed for 2D document representation where text pixels were encoded using character or word embeddings and classified into specific field types, using a convolutional neural network. GNN-based approaches (Liu et al., 2019a; Yu et al., 2021; Tang et al., 2021) adopted multi-modal features of text segments as nodes to model the document graph, and used graph neural networks to propagate information between neighboring nodes to attain a richer representation. In recent years, self-supervised pre-training has achieved great success. Inspired by the development of the pre-trained language models in various NLP tasks, recent studies on structured document pre-training (Xu et al., 2020, 2021a,b; Li et al., 2021a,b,c; Appalaraju et al., 2021) have pushed the limits. LayoutLM (Xu et al., 2020) modified the BERT (Devlin et al., 2019) architecture by adding 2D spatial coordinate embeddings. In comparison, our LiLT can be regarded as a more powerful and flexible solution for structured document understanding. LayoutLMv2 (Xu et al., 2021a) improved over LayoutLM by treating the visual fea-7753 Task Model Pre-training Docs FUNSD XFUND Avg. tures as separate tokens. Furthermore, additional pre-training tasks were explored to improve the utilization of unlabeled document data. SelfDoc (Li et al., 2021b) established the contextualization over a block of content, while StructuralLM (Li et al., 2021a) proposed cell-level 2D position embeddings and the corresponding pre-training objective. Recently, StrucTexT (Li et al., 2021c) introduced a unified solution to efficiently extract semantic features from different levels and modalities to handle the entity labeling and entity linking tasks. DocFormer (Appalaraju et al., 2021) designed a novel multi-modal self-attention layer capable of fusing textual, vision and spatial features. Nevertheless, the aforementioned SDU approaches mainly focus on a single language typically English, which is extremely limited with respect to multilingual application scenarios. To the best of our knowledge, LayoutXLM (Xu et al., 2021b) was the only pre-existing multilingual SDU model, which adopted the multilingual textual model InfoXLM (Chi et al., 2021) as the initialization, and adapted the LayoutLMv2 (Xu et al., 2021a) framework to multilingual structured document pre-training. However, it required a heavy process of multilingual data collection, cleaning and pre-training. On the contrary, our LiLT can deal with the multilingual structured documents by pre-training on the monolingual IIT-CDIP Test Collection 1.0 (Lewis et al., 2006) only. In this paper, we present LiLT, a language-independent layout Transformer that can learn the layout knowledge from monolingual structured documents and then generalize it to deal with multilingual ones. Our framework successfully first decouples the text and layout information in pre-training and then re-couples them for fine-tuning. Experimental results on eight languages under three settings (language-specific, cross-lingual zero-shot transfer, and multi-task fine-tuning) have fully illustrated its effectiveness, which substantially bridges the language gap in real-world structured document understanding applications. The public availability of LiLT is also expected to promote the development of document intelligence. For future research, we will continue to follow the pattern of transferring from monolingual to multilingual and further unlock the power of LiLT. In addition, we will also explore the generalized rather than language-specific visual information contained in multilingual structured documents. This research is (Grant No.: supported in part by NSFC 61936003) and GD-NSF (No. BERT-grid: Contextualized embedding for 2D document representation and understanding. In Workshop on Document Intelligence at NeurIPS . Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In NAACL-HLT , pages 41714186. ukasz Garncarek, Rafa Powalski, Tomasz Stanisawek, Bartosz Topolski, Piotr Halama, and Filip Gralinski. 2021. LAMBERT: Layout-aware (language) modeling using BERT for information extraction. In ICDAR . Adam W Harley et al. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. In ICDAR , pages 991995. Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. 2020. BROS: A pre-trained language model for understanding texts in document. Guillaume Jaume et al. 2019. FUNSD: A dataset for form understanding in noisy scanned documents. In ICDAR , volume 2, pages 16. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR . Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL-HLT , pages 260270. David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. 2006. Building a test collection for complex document information processing. In ACM SIGIR , pages 665 666. Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021a. StructuralLM: Structural pre-training for form understanding. In ACL . Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021b. SelfDoc: Self-supervised document representation learning. In CVPR , pages 5652 5660. 2017A030312006). References Muhammad Zeshan Afzal, Andreas Klsch, Sheraz Ahmed, and Marcus Liwicki. 2017. Cutting the error by half: Investigation of very deep CNN and advanced training strategies for document image classification. In ICDAR , volume 1, pages 883888. Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. DocFormer: End-to-end Transformer for document understanding. In ICCV . Dario Augusto Borges Oliveira et al. 2017. Fast CNN-based document layout analysis. In ICCV Workshop , pages 11731180. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Song-hao Piao, Ming Zhou, et al. 2020. UniLMv2: Pseudo-masked language models for unified language model pre-training. In ICML , pages 642652. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, He-Yan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In NAACL-HLT , pages 35763588. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmn, douard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL , pages 84408451. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Findings of EMNLP , pages 657668. Arindam Das, Saikat Roy, Ujjwal Bhattacharya, and Swapan K Parui. 2018. Document image classification with intra-domain transfer learning and stacked generalization of deep convolutional neural networks. In ICPR , pages 31803185. Tyler Dauphinee, Nikunj Patel, and Mohammad Rashidi. 2019. Modular multimodal architecture for document classification. arXiv preprint arXiv:1912.04376 .
abstain abstain objective abstain abstain other abstain abstain other abstain abstain abstain method result abstain objective objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other abstain other other other other other other other method other abstain method objective abstain abstain method objective other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other