Hierarchical neural architectures can efficiently capture long-distance dependencies and have been used for many document-level tasks such as summarization, document segmentation, and sentiment analysis. However, effective usage of such a large context can be difficult to learn, especially in the case where there is limited labeled data available. Building on the recent success of language model pretraining methods for learning flat representations of text, we propose algorithms for pre-training hierarchical document representations from unlabeled data. Unlike prior work, which has focused on pre-training contextual token representations or context-independent sentence/paragraph representations, our hierarchical document representations include fixed-length sentence/paragraph representations which integrate contextual information from the entire documents. Experiments on document segmentation, document-level question answering, and extractive document summarization demonstrate the effectiveness of the proposed pre-training algorithms.
In this work, the authors explore different ways to pre-train contextualized word and sentence representations for use in other tasks. They propose two main methods: a straight-forward extension of the ElMO model for hierarchical uni-directional language models, and a de-noising auto-encoder type method which allows to train bi-directional representations. The learned contextual representations are evaluated on three downstream tasks, demonstrating the superiority of the bi-directional training setting, and beating strong baselines on extractive summarization.
The method is clearly presented and easy to follow, and the experiments do seem to support the author's claims, but their exposition misses several important details (or could be presented more clearly). For the document segmentation task, are the articles taken from a held-out set, or are they contained in the pre-training set? For passage retrieval, is the representation the same or are the representations re-trained from scratch using paragraph blocks? What exactly are the other features (those can go in the appendix)? And for the extractive summarization task, how many sentences are selected? Is pre-training also done on Wikipedia, or are those representations trained on news text?
A comparison to non-contextualized sentence representations would also be welcome (SkipThought, InferSent, ElMO-pool for settings other than passage retrieval). Note also that the local pre-training is not equivalent to ElMO, as the later sees context form the whole document rather than just the current sentence.
It is interesting to see that contextualized sentence representations can be used and that the Mask-LM objective yields better results than L+R-LM, but these points would be better made if the above questions were answered.