Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) based model and has been the state-of-the-art for many kinds of Nature Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When we apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will decrease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length requires re-pretraining which will cost a mass of time and computing resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Arbitrarily Long Document Understanding) and it can accept arbitrarily long input without re-pretraining from scratch. We demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g., XLNet  and RoBERTa (Liu et al., 2019), for various NLP tasks with long text.
The paper proposes a methodology to overcome the problem of processing long sequences with a pre-trained Transformer model, which suffers from high computational costs due to the complexity being quadratic in the length of the sequence. The authors also point out that BERT needs to be retrained from scratch if sequences longer than the specified maximum length (512) are to be processed. Their method (BERT-AL) chunks the input text into segments of maximum length. Each segment is propagated through several layers of a pre-trained Transformer layer followed by a multi-channel LSTM. Herein, the positional embeddings in each segment are the same. The model is applied to extractive summarization, and directly compared to the BERTSUM model, which can only process documents up to length 512. The comparison is made for 4 application scenarios, each corresponding to an artificial maximum length after which the BERTSUM model truncates the input: after 8, 16, 128, and 256 tokens. The experimental results suggest that BERT-AL outperforms BERTSUM in all 4 scenarios: Substantially for max length 8 and 16, and marginally for 128 and 256. BERTSUM without any truncation still performs best, however.
I think the paper should be rejected for three main reasons: (1) The idea of using RNNs to overcome the long sequence problem in Transformers is already well studied. The specific proposed architecture may be new, but the design choices are not well justified. (2) The experimental evaluation is not convincing: The application scenarios are unrealistic, the dataset is not well-chosen, and the results are not impressive. (3) The structure and language of the paper needs to be polished.
Regarding (1): The problem of dealing with long sequences in Transformer models has been known for a long time, and there already exist several proposed solutions based on an RNN component, which are acknowledged in this paper. The authors propose another model for a similar purpose. While the model is interesting, the individual design decisions are not well-justified, e.g., why the LSTM is applied at each layer and why it needs to be a multi-channel LSTM. Neither a comparison with models from previous works nor an ablation study is provided, so that there is also no empirical justification for the model design. 
The authors argue that the novelty in their model comes from its applicability to pre-trained models, e.g., BERT and XLNET. While the motivation for BERT is reasonable, it is questionable whether this will still be a relevant for future research, as XLNET already mitigates the problem through the reliance on Transformer-XL, which is conceptually very similar to BERT-AL. If future pretrained models account for long sequences already during pre-training, the motivation for this work is rather low.
Regarding (2): In order to show the superiority of BERT-AL over BERTSUM, it would've been natural to employ it to datasets whose data are beyond what standard BERT can handle, i.e., longer than 512 tokens. Instead, the authors chose to evaluate on 4 rather unrealistic application scenarios derived from the CNN/DailyMail dataset, which artificially limit the range of the pre-trained BERT model to much less than what it can actually handle. While I understand the idea behind this setup, the insight that BERT performs poorly if the text is truncated to 8 or 16 tokens is not helpful. Since the improvement of BERT-AL over the baselines becomes marginal for longer sequences, and the performance of BERTSUM, without truncating the text, is not reached, I am not convinced of the model's value. An evaluation on more suitable datasets could help here.
Regarding (3): The paper is difficult to read for two reasons. First, the paper is not structured in a way that facilitates the understanding of the proposed method. For example, it is not clear why the pre-training tasks of BERT (masked language modeling and next sentence prediction) are relevant enough to be so prominently described. If the method is applicable to many pre-trained Transformers (as was claimed), then there is no need to go into this level of detail. On the same note, there is no need to explain BERTSUM's Inter-Sentence Transformer and Recurrent Neural Network variants for half a page if they are dismissed in the next paragraph. Second, the paper contains a lot of typos and grammatical mistakes. This is not a deal-breaker by itself, but it makes it obvious that the paper needs substantial polishing before it should be published. 
Questions and suggestions to the authors:
1) What is the advantage of choosing a multi-channel LSTM, where each segment is a channel? Wouldn't a single channel suffice that processes aggregated information from the segment (as the Transformer computes)? I think your paper would benefit from exploring different options like that experimentally to better justify your model decisions.
2) It is not clear to me how the information exchange between layers happens. If each layer consists of a single layer from a pre-trained Transformer followed by an LSTM, the input to the following layer of the pre-trained Transformer is the output of the LSTM, which is not part of the pre-trained model. That is, the following Transformer layer receives an input that is very different from what it has seen during pre-training, which we can not expect to work. If there is a misunderstanding on my side, could you please clarify that, or otherwise make it clearer in the paper?
3) I think a good baseline would be to do segment-wise encoding via the pre-trained Transformer, and then feed all sentence representations (i.e., the respective CLS tokens) from all segments into the Transformer-based summarization layer from BERTSUM. This would be a compromise between your model (doing segment-wise encoding but not using an LSTM) and BERTSUM, and thus could potentially highlight the importance of the LSTM.
4) I don't think showing the number of examples in your datasets as in Table 2 contributes much to the paper. Instead, I suggest to show statistics on the length of the documents / number of sentences etc., because these are directly relevant to your study.