Pretrained Language Models (LMs) have shown excellent results in achieving human-like performance on many language tasks. However, the most powerful LMs have one significant drawback: a fixed-sized input. Due to this constraint, these LMs are unable to utilize the full input of long documents. In this paper, we introduce a new framework to handle documents of arbitrary lengths. We investigate the addition of different mechanisms to handle an extended input size, and we demonstrate how attention can be used for parameter updates based on the most discriminating segments of input. We perform extensive validating experiments on patent and Arxiv datasets, both of which have long text. We show our method consistently outperforms state-of-the-art results reported in recent literature.
The paper presents a methods to combine multiple pre-trained language models using an attention mechanism in order to take the whole document into account. The effectiveness of the methods is evaluated on two long document classification tasks (Arxiv publication and patent classification) with state-of the art results.
Pros:
- the effectiveness of the methods is experimentally demonstrated using two relevant datasets
- the inverted wireless experiment clearly shows the interest of the attention combination strategy 
Cons:
- the methods is very simple (combining multiple segment prediction to perform document classifications) making the contribution of this paper quite weak. In the own terms of the authors, the "main contribution [...] is to investigate the effectiveness of different combination strategies". I am not sure that it is sufficient for the ICLR standards.
- the paper is difficult to read and should be proofread to improve readability
Minor issues:
- Socher et al (2011) uses recursive neural networks (ReNN) and not recurrent neural networks (RNN). RNN are ReNN but restrained to a linear chain structure
- In Introduction: ... in the  domain extremely complex data that is language ... -> I'm not sure the sentence is correct
- In Introduction: the last sentence should be shorten and rephrased.
- many typos