ERNIE DOC: The Retrospective Long Document Modeling Transformer
Transformers are not suited for processing long document input due to its quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or inferior modeling capability with comparable model size. In this paper, we propose ERNIEDOC, a document level language pretraining model based on Recurrence Transformers (Dai et al. 2019) Two well designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism enable ERNIE DOC with much longer effective context length to capture the contextual information of a whole document. We pretrain ERNIE DOC to explicitly learn the relationship among segments with an additional document aware segment reordering objective. Various experiments on both English and Chinese document level tasks are conducted. ERNIE DOC achieves SOTA language modeling result of 16.8 ppl on WikiText103 and outperforms competitive pretraining models on most language understanding tasks such as text classification, question answering by a large margin.
ACL/IJCNLP