Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images. While, there are still a large number of digital documents where the layout information is not fixed and needs to be interactively and dynamically rendered for visualization, making existing layout-based pre-training approaches not easy to apply. In this paper, we propose MarkupLM for document understanding tasks with markup languages as the backbone, such as HTML/XML-based documents, where text and markup information is jointly pre-trained. Experiment results show that the pre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding tasks. The pre-trained model and code will be publicly available at https:// aka.ms/markuplm . Multimodal pre-training with text, layout, and visual information has recently become the de facto approach (Xu et al., 2020, 2021a,b; Pramanik et al., 2020; Garncarek et al., 2021; Hong et al., 2021; Powalski et al., 2021; Wu et al., 2021; Li et al., 2021a,b; Appalaraju et al., 2021) in Visually-rich Document Understanding (VRDU) tasks. These multimodal models are usually pre-trained with the Transformer architecture (Vaswani et al., 2017) using large-scale unlabeled scanned document images (Lewis et al., 2006) or digital-born PDF files, followed by task-specific fine-tuning with relatively small-scale labeled training samples to achieve the state-of-the-art performance on a variety of document understanding tasks, including form understanding (Jaume et al., 2019; Xu et al., 2021b), receipt understanding (Huang et al., 2019;  Equal contributions during internship at Microsoft Research Asia. Corresponding authors: Lei Cui and Furu Wei Park et al., 2019), complex document understanding (Gralinski et al., 2020), document type classification (Harley et al., 2015), and document visual question answering (Mathew et al., 2021), etc. Significant progress has been witnessed not only in research tasks within academia, but also in different real-world business applications such as finance, insurance, and many others. Visually rich documents can be generally divided into two categories. The first one is the fixed-layout documents such as scanned document images and digital-born PDF files, where the layout and style information is pre-rendered and in-dependent of software, hardware, or operating system. This property makes existing layout-based pre-training approaches easily applicable to document understanding tasks. While, the second category is the markup-language-based documents such as HTML/XML, where the layout and style information needs to be interactively and dynamically rendered for visualization depending on the software, hardware, or operating system, which is shown in Figure 1. For markup-language-based documents, the 2D layout information does not exist in an explicit format but usually needs to be dynamically rendered for different devices, e.g., mobile/tablet/desktop, which makes current layout-based pre-trained models hard to apply. Therefore, it is indispensable to leverage the markup structure into document-level pre-training for downstream VRDU tasks. To this end, we propose MarkupLM to jointly pre-train text and markup language in a single framework for markup-based VRDU tasks. Distinct from fixed-layout documents, markup-based documents provide another viewpoint for the document representation learning through markup structures because the 2D position information and document image information cannot be used straightforwardly during the pre-training. Instead, MarkupLM takes advantage of the tree-based markup 6078 Similar to other multimodal pre-trained layout-based models, MarkupLM has four input embedding layers: (1) a text embedding that represents the token sequence information; (2) an XPath embedding that represents the markup tag sequence information from the root node to the current node; (3) a 1D position embedding that represents the sequence order information; (4) a segment embedding for downstream tasks. The overall architecture of MarkupLM is shown in Figure 2. The XPath embedding layer can be considered as the replacement of 2D position embeddings compared with the LayoutLM model family (Xu et al., 2020, 2021a,b). To effectively pre-train the MarkupLM, we use three pretraining strategies. The first is the Masked Markup Language Modeling (MMLM), which is used to jointly learn the contextual information of text and markups. The second is the Node Relationship Prediction (NRP), where the relationships are defined according to the hierarchy from the markup trees. The third is the Title-Page Matching (TPM), where the content within <title> ... </title> is randomly replaced by a title from another page to make the model learn whether they are correlated. In this way, MarkupLM can better understand the contextual information through both the language and markup hierarchy perspectives. We evaluate the MarkupLM models on the Web-based Structural Reading Comprehension (WebSRC) dataset (Chen et al., 2021) and the Structured Web Data Extraction (SWDE) dataset (Hao et al., 2011). Experiment results show that the pre-trained MarkupLM significantly outperforms the several strong baseline models in these tasks. The contributions of this paper are summarized as follows:  We propose MarkupLM to address the document representation learning where the layout information is not fixed and needs to be dynamically rendered. For the first time, the text and markup information is pre-trained in a single framework for the VRDU tasks. MarkupLM integrates new input embedding layers and pre-training strategies, which have been confirmed effective on HTML-based downstream tasks. The pre-trained MarkupLM models and codes for fine-tuning will be publicly available at https://aka.ms/markuplm . MarkupLM utilizes the DOM tree in markup language and the XPath query language to obtain the markup streams along with natural texts in markup-language-based documents (Section 2.1). We propose this Transformer-based model with a new XPath embedding layer to accept the markup sequence inputs (Section 2.2) and pre-train it with three different-level objectives, including Masked Markup Language Modeling (MMLM), Node Relation Prediction (NRP), and Title-Page Matching (TPM) (Section 2.3). A DOM 1 tree is the tree structure object of a markup-language-based document ( e.g., HTML or XML) in the view of DOM (Document Object Model) wherein each node is an object representing a part of the document. XPath 2 (XML Path Language) is a query language for selecting nodes from a markup-language-based document, which is based on the DOM tree and can be used to easily locate a node in the document. In a typical XPath expression, like /html/body/div/li[1]/div/span[2] , the texts stand for the tag name of the nodes while the subscripts are the ordinals of a node when multiple nodes have the same tag name under a common parent node. We show an example of DOM tree and XPath along with the corresponding source code in Figure 3, from which we can clearly identify the genealogy of all nodes within the document, as well as their XPath expressions. To take advantage of existing pre-trained models and adapt to markup-language-based tasks ( e.g. , webpage tasks), we use the BERT (Devlin et al., 2019) architecture as the encoder backbone and add a new input embedding named XPath embedding to the original embedding layer. The overview structures of MarkupLM and the newly-proposed XPath Embedding are shown in Figure 2 and 4. XPath Embedding For the i -th input token x i , we take its corresponding XPath expression 1 https://en.wikipedia.org/wiki/ Document_Object_Model 2 https://en.wikipedia.org/wiki/XPath <html> <head> <title> <body> <div> <li> Galaxy S20 <span> Release Date 2020 <li> <ul> <li> ... <span> Display <span> 6.5 inch <div> HTML Source Code DOM Tree & XPath <html><head><title>Galaxy S20 </title> </head><body><div><li><div> <span> Display </span> <span> 6.5 inch </span> </div> </li><li><div> <span> Processor </span> <span> Qualcomm Snapdragon </span> </div> </li><ul><li> <span> Release Date </span> 2020 </li> </ul> </div></body></html>  /html/body/div/li[1]/div/span[2]  XPath Extractor Figure 3: An example of DOM tree and XPath with the source HTML code. and split it by "/" to get the node information at each level of the XPath as a list, xp i = [( t i 0 , s i 0 ) , ( t i 1 , s i 1 ) ,    , ( t id , s id )] , where d is the depth of this XPath and ( t ij , s ij ) denotes the tag name and the subscript of the XPath unit on level j for x i . Note that for units with no subscripts, we assign 0 to s ij . To facilitate further processing, we do truncation and padding on xp i to unify their lengths as L . The process of converting XPath expression into XPath embedding is shown in Figure 4. For ( t ij , s ij ) , we input this pair into the j -th tag unit embedding table and j -th subscript unit embedding table respectively, and they are added up to get the j -th unit embedding ue ij . We set the dimensions of these two embeddings as d u . ue ij = TagUnitEmb j ( t ij )+ SubsUnitEmb j ( s ij ) We concatenate all the unit embeddings to get the intermediate representation r i of the complete XPath for x i . Finally, to match the dimension of other embeddings, we feed the intermediate representation r i into an FFN layer to get the final XPath embedding xe i . xe i = W 2 [ ReLU ( W 1 r i + b 1 )] + b 2 , W 1  R 4 d h  Ld u , b 1  R 4 d h , W 2  R d h  4 d h , b 2  R d h where d h is the hidden size of MarkupLM. To simplify the converting process, we have also tried replacing the FFN layer with a single linear transformation. However, this tiny modification makes the training process much more unstable and slightly hurts the performance so we keep the original design. To efficiently capture the complex structures of markup-language-based documents, we propose pre-training objectives on three different levels, including token-level (MMLM), node-level (NRP), and page-level (TPM). Masked Markup Language Modeling Inspired by the previous works (Devlin et al., 2019; Xu et al., 2020, 2021a), we propose a token-level pre-training objective Masked Markup Language Modeling (MMLM), which is designed to enhance the language modeling ability with the markup clues. Basically, with the text and markup input sequences, we randomly select and replace some tokens with [MASK] , and this task requires the model to recover the masked tokens with all markup clues. Node Relation Prediction Although the MMLM task can help the model improve the markup language modeling ability, the model is still not aware of the semantics of XPath information provided by the XPath embedding. With the naturally structural DOM tree, we propose a node-level pre-training objective Node Relation Prediction (NRP) to explicitly model the relationship between a pair of nodes. We firstly define a set of directed node relationships R  { self , parent , child , sibling , ancestor , descendent , others }. Then we combine each node to obtain the node pairs. For each pair of nodes, we assign the corresponding label according to the node relationship set, and the model is required to predict the assigned relationship labels with the features from the first token of each node. Title-Page Matching Besides the fine-grained information provided by markups, the sentence-level or topic-level information can also be leveraged in markup-language-based documents. For HTML-based documents, the element <title> can be excellent summaries of the <body> , which provides a supervision for high-level semantics. To efficiently utilize this self-supervised information, we propose a page-level pre-training objective Title-Page Matching (TPM). Given the element <body> of a markup-based document, we randomly replace 6081 the text of element <title> and ask the model to predict if the title is replaced by using the representation of token [CLS] for binary classification. We follow the scheme of common pre-trained language models (Devlin et al., 2019; Liu et al., 2019) and introduce the fine-tuning recipes on two downstream tasks including reading comprehension and information extraction. For the reading comprehension task, we model it as an extractive QA task. The question and context are concatenated together as the input sequence, and slicing is required when its length exceeds a threshold. For tokens of questions, the corresponding XPath embeddings are the same as [PAD] token. We input the last hidden state of each token to a binary linear classification layer to get two scores for start and end positions, and make span predictions with these scores following the common practice in SQuAD (Rajpurkar et al., 2016). For the information extraction task, we model it as a token classification task. We input the last hidden state of each token to a linear classification layer, which has n + 1 categories, where n is the number of attributes we need to extract and the extra category is for tokens that belong to none of these attributes. In this work, we apply our MarkupLM framework to HTML-based webpages, which is one of the most common markup language scenarios. Equipped with the existing webpage datasets Common Crawl (CC) 3 , we pre-train MarkupLM with large-scale unlabeled HTML data and evaluate the pre-trained models on web-based structural reading comprehension and information extraction tasks. Common Crawl The Common Crawl (CC) dataset contains petabytes of webpages in the form of raw web page data, metadata extracts, and text extracts. We choose one of its snapshots 4 , and use the pre-trained language detection model from fasttext (Joulin et al., 2017) to filter out non-English pages. Specifically, we only take the page when the model predicts it as English with the clas-sifier score > 0.6 and discard all the others. Besides, 3 https://commoncrawl.org/ 4 https://commoncrawl.org/2021/08/ july-august-2021-crawl-archive-available/ we only keep the tags that may contain texts ( e.g. <div> , <span> , <li> , <a> , etc.) and delete those with no texts ( e.g., <script> , <style> , etc.) in these pages to save storage space. After pre-processing, a subset of CC with 24M English webpages is extracted as our pre-training data for MarkupLM. WebSRC The Web-based Structural Reading Comprehension (WebSRC) dataset (Chen et al., 2021) consists of 440K question-answer pairs, which are collected from 6.5K web pages with corresponding HTML source code, screenshots, and metadata. Each question in WebSRC requires a certain structural understanding of a webpage to answer, and the answer is either a text span on the web page or yes/no. After adding the additional yes/no tokens to the text input, WebSRC can be modeled as a typical extractive reading comprehension task. Following the original paper (Chen et al., 2021), we choose evaluation metrics for this dataset as Exact match (EM) , F1 score (F1) , and Path overlap score (POS) . We use the official split to get the training and development set. Note that the authors of WebSRC did not release their testing set, so all our results are obtained from the development set. SWDE The Structured Web Data Extraction (SWDE) dataset (Hao et al., 2011) is a real-world webpage collection for automatic extraction of structured data from the Web. It involves 8 verticals, 80 websites (10 per vertical), and 124,291 webpages (200 2,000 per website) in total. The task is to extract the values corresponding to a set of given attributes (depending on which vertical the webpage belongs to) from a webpage, like value for author in book pages. Following previous works (Hao et al., 2011; Lin et al., 2020; Zhou et al., 2021), we choose page-level F1 scores as our evaluation metrics for this dataset. Since there is no official train-test split, we follow previous works (Hao et al., 2011; Lin et al., 2020; Zhou et al., 2021) to do training and evaluation on each vertical ( i.e. , category of websites) independently. In each vertical, we select k consecutive seed websites as the training data and use the remaining 10  k websites as the testing set. Note that in this few-shot extraction task, none of the pages in the 10  k websites have been visited in the training phase. This setting is abstracted from the real application scenario where only a small 6082 Model Modality EM F1 POS T-PLM (BERTBASE ) Text 52.12 61.57 79.74 H-PLM (BERTBASE ) Text + HTML 61.51 67.04 82.97 V-PLM (BERTBASE ) Text + HTML + Image 62.07 66.66 83.64 T-PLM (RoBERTa BASE ) Text 52.32 63.19 80.93 H-PLM (RoBERTa BASE ) Text + HTML 62.77 68.19 83.13 MarkupLM BASE Text + HTML 68.39 74.47 87.93 T-PLM (ELECTRALARGE ) Text 61.67 69.85 84.15 H-PLM (ELECTRALARGE ) Text + HTML 70.12 74.14 86.33 V-PLM (ELECTRALARGE ) Text + HTML + Image 73.22 76.16 87.06 T-PLM (RoBERTa LARGE ) Text 58.50 70.13 83.31 H-PLM (RoBERTa LARGE ) Text + HTML 69.57 74.13 85.93 MarkupLM LARGE Text + HTML 74.43 80.54 90.15 Table 1: Evaluation results on the WebSRC development set. set of labeled data is provided for specific websites and we aim to infer the attributes on a much larger unseen website set. The final results are obtained by taking the average of all 8 verticals and all 10 permutations of seed websites per vertical, leading to 80 individual experiments for each k . For the preand post-processing of data, we follow Zhou et al. (2021) to make a fair comparison. Pre-training The size of the selected tags and subscripts in XPath embedding are 216 and 1,001 respectively, the max depth of XPath expression ( L ) is 50, and the dimension for the tag-unit and subscript-unit embedding ( d u ) is 32. The token-masked probability in MMLM and title-replaced probability in TPM are both 15%, and we do not mask the tokens in the input sequence corresponding to the webpage titles. The max number of selected node pairs is 1,000 in NRP for each sample, and we limit the ratio of pairs with non-others ( i.e. , self, parent,    ) labels as 80% to make a balance. We initialize MarkupLM from RoBERTa and train it for 300K steps on 8 NVIDIA A100 GPUs. We set the total batch size as 256, the learning rate as 5e-5, and the warmup ratio as 0.06. The selected optimizer is AdamW (Loshchilov and Hutter, 2019), with (cid:15) = 1 e  6 ,  1 = 0 . 9 ,  2 = 0 . 98 , weight decay = 0 . 01 , and a linear decay learning rate scheduler with 6% warmup steps. We also apply FP16 , gradient-checkpointing (Chen et al., 2016), and deepspeed (Rasley et al., 2020) to reduce GPU memory consumption and accelerate training. Fine-tuning For WebSRC, we fine-tune MarkupLM for 5 epochs with the total batch size of 64, the learning rate of 1e-5, and the warmup ratio of 0.1. For SWDE, we fine-tune MarkupLM with 10 epochs, the total batch size of 64, the learning rate of 2e-5, and the warmup ratio of 0.1. The max sequence length is set as 384 in both tasks, and we keep other hyper-parameters as default. The results for WebSRC are shown in Table 1. Selected baselines are T-PLM, H-PLM, and V-PLM in Chen et al. (2021), referring to the paper for more details. To make a fair comparison, we re-run the released baseline experiments with RoBERTa. We observe MarkupLM significantly surpass H-PLM which uses the same modality of information. This strongly indicates that MarkupLM makes better use of the XPath features with the specially designed embedding layer and pre-training objectives compared with merely adding more tag tokens into the input sequence as in H-PLM. Besides, MarkupLM also achieves a higher score than the previous state-of-the-art V-PLM model that requires a huge amount of external resources to render the HTML source codes and uses additional vision features from Faster R-CNN (Ren et al., 2015), showing that our render-free MarkupLM is more lightweight and can learn the structural information better even without any visual information. It is also worth noting that adding HTML tags as input tokens in H-PLM and V-PLM drastically increases the length of input strings, so more slicing operations are required to fit the length limitation of language models, which results in more training samples (  860k) and longer training time, while MarkupLM does not suffer from this (only  470k training samples) and can greatly reduce training time. The results for SWDE are in Table 2 and 3. It is observed that our MarkupLM also substantially outperforms the strong baselines. Different from the previous state-of-the-art model SimpDOM which explicitly sends the relationship between DOM tree nodes into their model and adds huge amounts of extra discrete features ( e.g. , whether a node contains numbers or dates), MarkupLM is much simpler and is free from time-consuming additional webpage annotations. We also report detailed statistics with regard to different verticals in Table 3. With the growth of k , MarkupLM gets more webpages as the training set, so there is a clear ascending trend reflected by the scores. We also see the variance among different verticals since the number and type of pages are not the same. To investigate how each pre-training objective contributes to MarkupLM, we conduct an ablation study on WebSRC with a smaller training set containing 1M webpages. The model we initialized from is BERT-base-uncased in this sub-section with all the other settings unchanged. The results are in Table 4. According to the four results in #1, we see both of the newly-proposed training objectives improve the model performance substantially, and the proposed TPM (+4.6%EM) benefits the model more than NRP (+2.4%EM). Using both objectives together is more effective than using either one alone, leading to an increase of 5.3% on EM. We can also see a performance improvement (+1.9%EM) from #1d to #2a when replacing BERT with a stronger initial model RoBERTa. Finally, we get the best model with all three objectives and better initialization on larger data, as the comparison between #2a and #2b. Multimodal pre-training with text, layout, and image information has significantly advanced the research of document AI, and it has been the de facto approach in a variety of VRDU tasks. Although great progress has been achieved for the fixed-layout document understanding tasks, the existing multimodal pre-training approaches cannot be easily applied to markup-based document understanding in a straightforward way, because the layout information of markup-based documents needs to be rendered dynamically and may be different depending on software and hardware. Therefore, the markup information is vital for the document understanding. Ashby and Weir (2020) compared the Text+Tags approach with their Text-Only equivalents over five web-based NER datasets, which indicates the necessity of markup enrichment of deep language models. Lin et al. (2020) presented a novel two-stage neural approach named FreeDOM. The first stage learns a representation for each DOM node in the page by combining both the text and markup information. The second stage captures longer range distance and semantic relatedness using a relational neural network. Experiments show that FreeDOM beats the previous SOTA results without requiring features over rendered pages or expensive hand-crafted features. Zhou et al. (2021) proposed a novel transferable method SimpDOM to tackle the problem by efficiently retrieving useful context for each node by leveraging the tree structure. Xie et al. (2021) introduced a framework called WebKE that extracts knowledge triples from semi-structured webpages by extending pre-trained language models to markup language and encoding layout semantics. However, these methods did not fully leverage the large-scale unlabeled data and self-supervised pre-training techniques to enrich the document representation learning. To the best of our knowledge, MarkupLM is the first large-scale pre-trained model that jointly learns the text and markup language in a single framework for VRDU tasks. In this paper, we present MarkupLM, a simple yet effective pre-training approach for text and markup language. With the Transformer architecture, MarkupLM integrates different input embeddings including text embeddings, positional embeddings, and XPath embeddings. Furthermore, we also propose new pre-training objectives that are specially designed for understanding the markup language. We evaluate the pre-trained MarkupLM model on the WebSRC and SWDE datasets. Experiments show that MarkupLM significantly outperforms several SOTA baselines in these tasks. For future research, we will investigate the MarkupLM pre-training with more data and more computation resources, as well as the language expansion. Furthermore, we will also pre-train MarkupLM models for digital-born PDFs and Of-fice documents that use XML DOM as the backbones. In addition, we will also explore the relationship between MarkupLM and layout-based models (like LayoutLM) to deeply understand whether these two kinds of models can be pre-trained under a unified multi-view and multi-task setting and whether the knowledge from these two kinds of models can be transferred to each other to better understand the structural information.
abstain abstain objective abstain other other other abstain abstain abstain abstain other abstain objective abstain abstain abstain abstain method abstain abstain abstain abstain method abstain objective objective objective objective abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain result abstain abstain abstain abstain result abstain other other other other other other other other other other other objective method abstain objective method abstain objective method objective