Transformers are not suited for processing long documents, due to their quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or lead to an inferior modeling capability against comparable model sizes. In this paper, we propose ERNIE-DOC , a document-level language pretraining model based on Recurrence Transformers (Dai et al., 2019). Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism, enable ERNIE-DOC 1 , which has a much longer effective context length, to capture the contextual information of a complete document. We pretrain ERNIE-DOC to explicitly learn the relationships among segments with an additional document-aware segment-reordering objective. Various experiments were conducted on both English and Chinese document-level tasks. ERNIE-DOC improved the state-of-the-art language modeling result of perplexity to 16.8 on WikiText-103. Moreover, it outperformed competitive pretraining models by a large margin on most language understanding tasks, such as text classification and question answering. Transformers (Vaswani et al., 2017) have achieved remarkable improvements in a wide range of natural language tasks, including language modeling (Dai et al., 2019), text classification (Yang et al., 2019), and question answering (Devlin et al., 2018; Radford et al., 2019). This success is largely due to the self-attention mechanism, which enables the network to capture contextual information from the *indicates equal contribution. 1 Source code and pre-trained checkpoints can be found at https://github.com/PaddlePaddle/ERNIE/ tree/repro/ernie-doc . entire input sequence. Nevertheless, the memory usage and computation complexity caused by the self-attention mechanism grows quadratically with the sequence length, incurring excessive cost when processing a long document on existing hardware. Currently, the most prominent pretrained models, such as BERT (Devlin et al., 2018), are used on fixed-length input segments of a maximum of 512 tokens owing to the aforementioned limitation. Thus, a long document input must be partitioned into smaller segments of manageable sizes. However, this leads to the loss of important cross-segment information, that is, the context fragmentation problem (Dai et al., 2019), as shown in Fig. 1(a). To mitigate the problem of insufficient interactions among the partitioned segments of long documents, Recurrence Transformers (Dai et al., 2019; Rae et al., 2019) permit the use of contextual information from previous segments in computing the hidden states for a new segment by maintaining a memory component from the previous activation; this enables the modeling of long documents. In addition, Sparse Attention Transformers (Child et al., 2019; Tay et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020) focus on reducing the complexity of self-attention operations to explicitly improve the modeling length, but only up to a restricted context length (4,096) due to resource limitations. We argue that existing strategies are not sufficiently effective or reliable, because the contextual information of a complete document is still not available for each segment during the training phase . As depicted in Fig. 1, when training on segment S 2 , the model is ideally optimized by maximizing P ( y | ( S 1 , S 2 , S 3 )) conditioned on the contextual information of the entire document D = { S 1 , S 2 , S 3 } , in contrast to the following suboptimal solutions: P ( y | S 2 ) for Vanilla/Sparse Transformers 2 and P ( y | ( S 1 , S 2 )) for Recurrence Transformers. To address this limitation, we propose ERNIE-DOC (A Retrospective Long-Document Modeling Transformer) based on the Recurrence Transformer paradigm. Inspired by the human reading behavior of skimming a document first and then looking back upon it attentively, we design a retrospective feed mechanism in which segments from a document are fed twice as input. As a result, each segment in the retrospective phase could explicitly fuse the semantic information of the entire document learned in the skimming phase, which prevents context fragmentation. However, simply incorporating the retrospective feed mechanism into Recurrence Transformers is infeasible because the maximum effective context length is limited by the number of layers (Dai et al., 2019), as shown in Fig. 1 (b). Thus, we present an enhanced recurrence mechanism , a drop-in replacement for a Recurrence Transformer, by changing the shifting-one-layer-downwards recurrence to the same-layer recurrence. In this manner, the maximum effective context length can be expanded, and past higher-level representations can be exploited to enrich future lower-level representations. Moreover, we introduce a segment-reordering objective to pretrain a document-level model. Specifically, it is a document-aware task of predicting the correct order of the permuted set of segments of a document, to model the relationship among segments directly. This allows ERNIE-2 For Sparse Transformers, the length of segment S 2 could be up to 4,096 in Beltagy et al. (2020); Zaheer et al. (2020). DOC to build full document representations for prediction. This is analogous to the sentence-reordering task in ERNIE 2.0 (Sun et al., 2020b) but at a segment level of granularity, spanning (commonly) multiple training steps. We first evaluate ERNIE-DOC on autoregressive word-level language modeling using the enhanced recurrence mechanism, which, in theory, allows the model to process a document with in-finite words. ERNIE-DOC achieves state-of-the-art (SOTA) results on the WiKiText-103 benchmark dataset, demonstrating its effectiveness in long-document modeling. Then, to evaluate the potential of ERNIE-DOC on document-level natural language understanding (NLU) tasks, we pretrained the English ERNIE-DOC on the text corpora utilized in BigBird (Zaheer et al., 2020) from the RoBERTa-released checkpoint, and the Chinese ERNIE-DOC on the text corpora utilized in ERNIE 2.0 (Sun et al., 2020b) from scratch. After pretraining, we fine-tuned ERNIE-DOC on a wide range of English and Chinese downstream tasks, including text classification, question answering and keypharse extraction. Empirically, ERNIE-DOC consistently outperformed RoBERTa on various benchmarks and showed significant improvements over other high-performance long-text pretraining models for most tasks. Sparse Attention Transformers have been extensively explored (Child et al., 2019; Tay et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020). The key idea is to sparsify the self-attention operation, which scales quadratically with the sequence length. For instance, the Sparse Transformer (Child et al., 2019) uses a dilated sliding window that reduces the complexity to O ( L  L ) , where L is the sequence length. Reformer (Kitaev et al., 2020) further reduces the complexity to O ( L log L ) using locality-sensitive hashing attention to compute the nearest neighbors. BP-Transformers (Ye et al., 2019) employs a binary partition for the input sequence. Recently, Longformer (Beltagy et al., 2020) and BigBird (Zaheer et al., 2020) have been proposed, and both achieved state-of-the-art performance on a variety of long-document tasks. They reduce the complexity of self-attention to O ( L ) by combining random attention, window attention, and global attention. However, it has been proven in Zaheer et al. (2020) that sparse attention mechanisms cannot universally replace dense attention mechanisms; moreover, solving the simple problem of finding the furthest vector requires ( n ) -layers of a sparse attention mechanism but only O (1) layers of a dense attention mechanism. In addition, the aforementioned methods require customized CUDA kernels or TVM programming to implement sparse attention, which are not maintainable and are difficult to use. In this study, we adopt a different approach to adapting Recurrence Transformers for a pretraining-then-finetuning setting, to model a long document. Recurrence Transformers (Dai et al., 2019; Rae et al., 2019) have been successfully applied in generative language modeling. They employ the Transformer decoder as a parametric model for each conditional distribution in p ( x ) = (cid:81) Lt =1 p ( x t | x <t ) , where x denotes a text sequence. To capture long dependencies, they process the text in segments from left to right based on the segment recurrence mechanism (Dai et al., 2019). This mechanism maintains a memory bank of past activations at each layer to preserve a history of context. Compressive Transformer (Rae et al., 2019) adds a compressive memory bank to sufficiently store old activations instead of discarding them, which facilitates long-range sequence learning. However, these methods operate from left to right, which limits their capacity for discriminative language understanding tasks that require bidirectional information. XLNet (Yang et al., 2019) proposed a permutation language modeling objective to construct bidirectional information and achieve supe-rior performance in multiple NLP tasks; however, its application to long-document modeling tasks remains largely unexplored. ERNIE-DOC builds on the ideas of the Recurrence Transformers to 1) tackle the limitation of Recurrence Transformers for utilizing bidirectional contextual information and 2) improve the behavior of the segment recurrence mechanism to capture longer dependencies. Hierarchical Transformers (Zhang et al., 2019; Lin et al., 2020) have enabled significant progress on numerous document-level tasks, such as document summarization (Zhang et al., 2019) and document ranking (Lin et al., 2020). Similar to Vanilla Transformers, Hierarchical Transformers also split long documents into shorter segments with manageable lengths and then feed them independently to produce corresponding segment-level semantic representations. Unlike in Vanilla Transformers, however, separate Transformer layers are used in Hierarchical Transformers to process the concatenation of these representations. Hierarchical Transformers ignore the contextual information from the remaining segments when processing each segment of a long document, thus suffering from the context fragmentation problem. In this section, we first describe the background (Sec. 3.1) that ERNIE-DOC builds on. Then, we present the implementation of ERNIE-DOC , including the retrospective feed mechanism in Sec. 3.2, the enhanced recurrence mechanism in Sec. 3.3, and the segment-reordering objective in Sec. 3.4. Formally, a long document D is sliced into T sequential segments, denoted as { S 1 , S 2 , ..., ST } , where S  = { x , 1 , x , 2 , ..., x ,L } is the  -th segment with L tokens; x denotes a single token. Vanilla, Sparse, and Recurrence Transformers employ different strategies to produce the hidden state h n  RL  d for segment S  at the n -th layer: (cid:101) h n  1  +1 = (cid:40) h n  1  +1 , Vanilla or Sparse Transformers [ SG ( h n  1  )  h n  1  +1 ] , Recurrence Transformers , q n +1 , k n +1 , v n +1 = h n  1  +1 W (cid:62) q , (cid:101) h n  1  +1 W (cid:62) k , (cid:101) h n  1  +1 W (cid:62) v . h n +1 = Transformer-Block ( q n +1 , k n +1 , v n +1 ) . (1) where q  RL  d , k , and v  R ( L + m )  d are the query, key and value vectors, respectively with hidden dimension d and memory length m (Note that m = 0 for Vanilla or Sparse Transform-ers); (cid:101) h n  1  +1  R ( L + m )  d is the extended context; W   R d   d represents learnable linear projection parameters; the function SG (  ) denotes the stop-gradient operation; and the notation [  ] denotes the concatenation of two hidden states along the length dimension. In contrast to Vanilla or Sparse Transformers, where h n +1 is produced using only itself, Recurrence Transformers introduce a segment-level recurrence mechanism to promote interaction across segments. The hidden state computed for the previous segment h n  1  is cached as an auxiliary context to help process the current segment h n . However, from the concatenation part in Eq. 1, i.e., [ SG ( h n  1  )  h n  1  +1 ] , there is apparently a constraint that the current hidden state can only fuse information from the previous segments. In S 1 S 2 S 3 S 4 Recurrence Transformers S 1 S 2 S 3 S 4 Larger E  ective Context Length S 1 S 2 S 3 S 4 ERNIE-DOC Transformer block Memory concatenation Hidden states input E  ective Context Larger E  ective Context Length Retrospective Phase Layer-1 Layer-2 Layer-3 Layer-1 Layer-2 Layer-3 The Retrospective Phase Figure 2: Illustrations of ERNIE-DOC and Recurrence Transformers, where models with three layers take as input a long document D which is sliced into four segments S i , i  [1 , 2 , 3 , 4] . ERNIE-DOC employs a retrospective feed mechanism to address the unavailability of the contextual information of a complete document for each segment . The segments from a long document are twice fed as input. Mimicking the human reading behavior, we refer to the first and second input-taking phases as the skimming and retrospective phases, respectively. In the skimming phase, we employ a recurrence mechanism to cache the hidden states for each segment. In the retrospective phase, we reuse the cached hidden states from the skimming phase to enable bi-directional information flow. Naively, we can rewrite Eq. 1 to obtain the contextual information of an entire document in the skimming phase to be utilized in the retrospective phase as follows, (cid:98) H = [ (cid:98) H 11: T  (cid:98) H 21: T     (cid:98) HN 1: T ] , (skim. phase) (cid:101) h n  1  +1 = [ SG ( (cid:98) H  h n  1  )  h n  1  +1 ] , (retro. phase) (2) where (cid:98) H  R ( L  T  N )  d denotes the cached hidden states in the skimming phase with T segments, L length of each segment and total N layers, and (cid:98) H i 1: T = [ (cid:98) h i 1  (cid:98) h i 2     (cid:98) h iT ] is the concatenation of i -th layer's hidden states of the skimming phase. Thus, the extended context (cid:101) h n  1  +1 is guaranteed to capture the bidirectional contextual information of the entire document. However, it will incur massive memory and computation cost for directly employing (cid:98) H in self-attention mechanism. Henceforth, the main issue is how (cid:98) H should be implemented in a memoryand computation-efficient manner. By rethinking segment-level recurrence (Dai et al., 2019), we observe that the largest possible context dependency length increases linearly w.r.t the number of layers ( N ). For instance, at i -th layer, (cid:98) h i have the longest dependency to (cid:98) h 1   ( i  1) . Thus, to minimize memory and computation consumption, hidden states from the N -th layer (top-layer) are included at a stride of N , which is suf-ficient to build the contextual information of an entire document. Formally, (cid:98) H can be reduced to (cid:98) H r = [ (cid:98) h NN  (cid:98) h N 2  N    (cid:98) h N (cid:98) T/N (cid:99) N ] (Note that when T is not evenly divisible by N , the last hidden state (cid:98) h NT need to be included). However, for a long document input, the extra computational and memory cost of (cid:98) H r  R (cid:100) T/N (cid:101) d where T (cid:29) N is still excessive on existing hardware. To effectively utilize the retrospective feed mechanism in practice, an ideal strategy is to ensure that the cached hidden state h n  1  already contains the contextual information of an entire document without explicitly taking (cid:98) H or (cid:98) H r as input. Essentially, we should tackle the problem of limited effective context length in the segment-level recurrence mechanisms. Herein, we introduce the enhanced recurrence mechanism, a drop-in replacement for the segment-level recurrence mechanism, by changing the shifting-one-layer-downwards recurrence to the same-layer recurrence as follows: (cid:101) h n  1  +1 = [ SG ( h n )  h n  1  +1 ] (3) where the cached hidden state h n  1  in Eq. 1 and Eq. 2 is replaced with h n in Eq. 3. As shown in Fig. 2, when the retrospective feed mechanism is combined with the enhanced recurrence mechanism, every segment in the retrospective phase (shown in the box with a green dotted border) has bidirectional contextual information of the entire text input. We successfully modeled a larger effective context length (shown in the box with a orange dotted border) than traditional Recurrence Transformers can without extra memory and computation costs. Another benefit of the enhanced recurrence scheme is that past higher-level representations can be exploited to enrich future lower-level representations. In addition to the masked language model (MLM) objective (Devlin et al., 2018), we introduce an additional document-aware task called segment-reordering objective for pretraining. Benefitting from the much larger effective context length provided by the enhanced recurrence mechanism, the goal of the segment-reordering objective is to predict the correct order for the permuted set of segments of a long document, to explicitly learn the relationships among segments. During the pretraining process of this task, a long text input D is first randomly partitioned into 1 to m chunks; then, all the combinations are shuffled in a random order. As shown in Fig. 3, D is partitioned into three chunks and then permuted, that is, D = { C 1 , C 2 , C 3 } =   D = { C 2 , C 3 , C 1 } , where C i denotes the i -th chunk. Subsequently, the permuted long context  D is split into T sequential segments as a common practice, denoted as  D = { S 1 , S 2 , ..., ST } . We let the pretrained model reorganize these permuted segments, modeled as a K -class classification problem, where K = (cid:80) mi =1 i ! . The pretraining objective is summarized as follows for the  -th input segment: max  log p  ( S  |  S  ) + 1  = T log p  ( D|  D ) where  S  is the corrupted version of S  , which is obtained by randomly setting a portion of tokens S 1 S 2 ST Segments (~512 tokens each) (cid:51)(cid:72)(cid:85)(cid:80)(cid:88)(cid:68)(cid:87)(cid:72)(cid:71)(cid:3)(cid:38)(cid:75)(cid:88)(cid:81)(cid:78)(cid:86)(cid:3)(cid:82)(cid:73) (cid:68)(cid:3)(cid:47)(cid:82)(cid:81)(cid:74)(cid:3)(cid:55)(cid:72)(cid:91)(cid:87)(cid:3)(cid:44)(cid:81)(cid:83)(cid:88)(cid:87) Model label =  C 1 C 2 C 3  C 2 : [Related Work] Sparse attention based transformers are largely explored  C 3 : [Proposed Method] In this section, we firstly describe the background of proposed ERNIE-DOC  C 1 : [Introduction] Transformers have achieved remarkable improvements  (cid:335) (cid:335) Figure 3: Illustrations of segment-reordering objective. to [MASK] ;  D is the permutated version of D ;  is the model parameter; and 1  = T indicates that the segment-reordering objective is optimized only at the T -th step. Autoregressive language modeling aims to estimate the probability distribution of an existing to-ken/character based on previous tokens/characters in an input sequence. For comparison with previous work, we conducted experiments on word-level LM, that is, WikiText-103 (Merity et al., 2016), which is a document-level language modeling dataset. For autoregressive language modeling, we use a memory-enhanced Transformer-XL (Dai et al., 2019), that is, we employ our enhanced recurrence mechanism to replace the primitive one used in the Transformer-XL. Additionally, as proposed by Segatron (Bai et al., 2020), we introduce the segment-aware mechanism into Transformer-XL. Based on Transformer-XL, we trained a base-size model (L=16, H=410, A=10) and a large-size model (L=18, H=1,024, A=16) 3 . The models were trained for 200K/400K steps using a batch size of 64/128 for the base/large configurations. During the training phase, the sequence length and memory length were limited to 150 and 384 for the base and the large model, respectively. The remaining hyper-parameters were identical to those of Transformer-XL. 3 We denote the number of Transformer layers as L, the hidden size as H, and the number of self-attention heads as A. Models #Param. Tab. 1 summarizes the evaluation results for WikiText-103. ERNIE-DOC achieves an impressive improvement compared with Transformer-XL: the perplexity (PPL) decreases by 3.0 for the base model and by 1.5 for the large model. Finally, we improve the state-of-the-art result of PPL to 21.0 (the base model) and 16.8 (the large model). English Data. To allow ERNIE-DOC to capture long dependencies in pretraining, we compiled a corpus from four standard datasets: WIKIPEDIA , BOOKSCORPUS (Zhu et al., 2015), CC-NEWS 4 , and STORIES (Trinh and Le, 2018) (details listed in Tab. 2). We tokenized the corpus using the RoBERTa wordpieces tokenizer (Liu et al., 2019) and duplicated the pretraining data 10 times. Chinese Data. The Chinese text corpora used in ERNIE 2.0 (Sun et al., 2020b) were adopted for pretraining ERNIE-DOC . Pretraining. We trained three sizes of models for English tasks: small (L=6, H=256, A=4), base (L=12, H=768, A=12), and large (L=24, H=1,024, 4 We used news-please to crawl English news articles published between September 2016 and February 2019 and adopted Message Digest Algorithm5 (MD5) for deduplication. A=16). For Chinese tasks, we used only one size, i.e., base (L=12, H=768, A=12). We limited the length of the sentences in each mini-batch to 512 tokens and the length of the memory to 128. The models were trained for 500K/400K/100K steps using a batch size of 2,560/2,560/3,920 sentences for the small/base/large configurations. ERNIE-DOC was optimized with the Adam (Kingma and Ba, 2014) optimizer. The learning rate was warmed up over the first 4,000 steps to a peak value of 1e-4, and then it linearly decayed. The remaining pretraining hyperparameters were the same as those of RoBERTa (Liu et al., 2019) (see Tab. 12). Additionally, we employed relative positional embedding (Shaw et al., 2018) in our model pretraining because it is necessary for reusing hidden state without causing temporal confusion (Dai et al., 2019). Finetune. In contrast to previous models, such as BERT, RoBERTa, and XLNet, the proposed model employs the retrospective feed mechanism and the enhanced recurrence mechanism during the finetuning phase to fully utilize the advantages of these two strategies. Results on Long-Text Classification Tasks . We consider two datasets: IMDB reviews (Maas et al., 2011) and Hyperpartisan News Detection (HYP) (Kiesel et al., 2019). The former is a widely used sentiment analysis dataset containing 50,000 movie reviews, labeled as positive or negative. The latter contains news that takes extreme left-wing or right-wing standpoints. The documents in HYP are extremely long (50% of the samples contain more than 537 tokens) and are thus suitable for testing long-text classification ability. Tab. 3 summarizes the results of the ERNIE-DOC -Base and ERNIE-DOC -Large models for long-text classification tasks, and ERNIE-DOC achieves a SOTA result. On IMDB, we observed a modest perfor-Models TQA HQA F1 Span Supp Joint RoBERTa 74.3 73.5 83.4 63.5 Longformer 75.2 74.3 84.4 64.4 BigBird 79.5 75.5 87.1 67.8 ERNIE-DOC 80.1 79.4 86.3 70.5 Longformer-Large 77.8 81.0 85.8 71.4 BigBird-Large -81.3 89.4 -ERNIE-DOC -Large 82.5 82.2 87.6 73.7 Table 4: Results on TQA and HQA dev dataset for document-level QA. mance gain compared with RoBERTa. This is because nearly 90% of the samples in the dataset consist of fewer than 569 tokens. Unlike on IMDB, ERNIE-DOC surpasses the baseline models on HYP by a substantial margin, demonstrating its capability of utilizing information from a long document input. Note that we include XLNet-Large, the previous SOTA pretraining model on the IMDB dataset, as the baseline for a large model setting; ERNIE-DOC achieves a result comparable to that of XLNet-Large. Results on Document-level Question-Answering Tasks . We utilized two document-level QA datasets (Wikipedia setting of TriviaQA (TQA) (Joshi et al., 2017) and distractor setting of HotpotQA (HQA) (Yang et al., 2018)) to evaluate the reasoning ability of the models over long documents. TQA and HQA are extractive QA tasks, and we follow the simple QA model of BERT (Devlin et al., 2018) to predict an answer with the maximum sum of start and end logits across multiple segments of a sample. In addition, we use a modified cross-entropy loss (Clark and Gardner, 2017) for the TQA dataset and use a two-stage model (Groeneveld et al., 2020) with the backbone of ERNIE-DOC for the HQA dataset. Tab. 4. shows that ERNIE-DOC outperforms RoBERTa and Longformer by a considerable margin on these two datasets, and is comparable to current SOTA long-document model, i.e., BigBird on HQA in large-size model setting. Results on the Keyphrase Extraction Task . We include OpenKP (Xiong et al., 2019) dataset to evaluate ERNIE-DOC 's ability to extract keyphrases from a long document. Each document contains up to three short keyphrases and we follow the model setting of JointKPE (Sun et al., 2020a) and ETC (Ainslie et al., 2020) by applying CNNs on BERT's output to compose n-gram embeddings for classification. We report the results of base-size models in Tab. 5 under no-visual-features setting for easy and fair comparison with baselines. ERNIE-DOC performs stably better on all metrics on the OpenKP dataset. We conducted extensive experiments on seven Chinese natural language understanding (NLU) tasks, including machine reading comprehension (CMRC2018 (Cui et al., 2018), DRCD (Shao et al., 2018), DuReader (He et al., 2017), C 3 (Sun et al., 2019a)), semantic similarity (CAIL2019-SCM (CAIL) (Xiao et al., 2019)), and long-text classification (IFLYTEK (IFK) (Xu et al., 2020), THUCNews (THU) 5 (Sun et al., 2016)). The documents in all the aforementioned datasets are sufficiently long to be used to evaluate the effectiveness of ERNIE-DOC on long-context tasks (see detailed datasets statistics in Tab. 9). We reported the mean results with five runs for the seven Chinese tasks in Tab. 6, and summarized the hyperparameters in Tab. 16. ERNIE-DOC outperforms previous models across these Chinese NLU tasks by a significant margin in the base-size model group. 4.2.5 Ablation Studies No. No.IV and No.V, we see that segment-level recurrence is necessary for modeling long documents and produces 2.74 and 3.95 % points improvement on the TQA and HYP dateset, respectively. Moreover, a substantial improvement is achieved using the enhance recurrence mechanism (2.29% point on TQA and 1.40% point on HYP, see No.III IV). Retrospective feed mechanism further improves 0.21% point on TQA and 1.33% point on HYP (No.II No.III). Considering different types of tasks, we observe that on HYP, an extremely long text classification dataset, a substantial improvement is achieved using the segment-reordering objective (1.5% point). This indicates that the [CLS] token, pretrained using the segment-reordering objective, is more adaptable to the document-level text classification task. Effect of enhanced recurrence mechanism with regard to different maximum sequence lengths . As depicted in Fig. 4, the enhanced recurrence mechanism plays an important role in pretraining an effective language model with lower PPL and higher accuracy under both the maximum sequence input lengths of 128 and 512. The effect of the enhanced recurrence mechanism is more significant under a smaller maximum sequence length, even makes the ERNIE-DOC -Small (max-len:128) comparable to ERNIE-DOC -Small w/o en recur (max-len:512) w.r.t accuracy. This intriguing property of the enhanced recurrence mechanism enables more efficient model training and inference by reducing maximum sequence length while remaining comparable modeling capability. In this paper, we proposed ERNIE-DOC , a document-level language pretraining model based on the Recurrence Transformers paradigm. Two well-designed mechanisms, namely the retrospective feed mechanism and the enhanced recurrent mechanism, enable ERNIE-DOC , which theoretically has the longest possible dependency, to model bidirectional contextual information of a complete document. Additionally, ERNIE-DOC is pretrained with a document-aware segment-reordering objective to explicitly learn the relationship among segments of a long context. Experiments on various downstream tasks demonstrate that ERNIE-DOC outperforms existing strong pretraining models such as RoBERTa, Longformer, and BigBird and achieves SOTA results on several language modeling and language understanding benchmarks. In future studies, we will evaluate ERNIE-DOC on language generation tasks, such as generative question answering and text summarization. We will also investigate its potential applicability in other areas, such as computational biology. Another possibility is to incorporate graph neural networks into ERNIE-DOC to enhance its modeling capability for tasks that require multi-hop reasoning and long-document modeling ability. This work was supported by the National Key Research and Development Project of China (No. 2018AAA0101900).
abstain abstain abstain abstain objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other objective objective abstain abstain abstain method abstain objective abstain abstain abstain abstain objective abstain method abstain abstain other other other other other other other other other method other other other other other other other other other other other other method method other other other other other other other other other abstain abstain method method method method other abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method objective abstain other