BERT-AL: BERT for Arbitrarily Long Document Understanding
Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) based model and has been the state-of-the-art for many kinds of Nature Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When we apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will decrease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length requires re-pretraining which will cost a mass of time and computing resources. What*s even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Arbitrarily Long Document Understanding) and it can accept arbitrarily long input without re-pretraining from scratch. We demonstrate BERT-AL*s effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g., XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), for various NLP tasks with long text.
The author proposes an extended version of the BERT architecture, BERTAL for text summarization. BERTAL aims to overcome the limit of maximal allowable text length in BERT, and the experiments show some consistent performance enhancement to baseline architecture using BERT and approaching performance to state-of-the-art architecture using BERT (BERTSUM), where BERTSUM has the maximal length limit . The experimental procedures as well as the choice of architectural design are well explained and designed in a logical way. Substantial comparison experiments also pinpoint the performance. However, it doesn’t compare the result of BERTAL to other text summarizers with arbitrary-length text, which is not based on BERT.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
BERT-AL: BERT for Arbitrarily Long Document Understanding
Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) based model and has been the state-of-the-art for many kinds of Nature Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When we apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will decrease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length requires re-pretraining which will cost a mass of time and computing resources. What*s even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Arbitrarily Long Document Understanding) and it can accept arbitrarily long input without re-pretraining from scratch. We demonstrate BERT-AL*s effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g., XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), for various NLP tasks with long text.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The author proposes an extended version of the BERT architecture, BERTAL for text summarization. BERTAL aims to overcome the limit of maximal allowable text length in BERT, and the experiments show some consistent performance enhancement to baseline architecture using BERT and approaching performance to state-of-the-art architecture using BERT (BERTSUM), where BERTSUM has the maximal length limit . The experimental procedures as well as the choice of architectural design are well explained and designed in a logical way. Substantial comparison experiments also pinpoint the performance. However, it doesn’t compare the result of BERTAL to other text summarizers with arbitrary-length text, which is not based on BERT.
assistant
The author proposes an extended version of the BERT architecture, BERTAL for text summarization. BERTAL aims to overcome the limit of maximal allowable text length in BERT, and the experiments show some consistent performance enhancement to baseline architecture using BERT and approaching performance to state-of-the-art architecture using BERT (BERTSUM), where BERTSUM has the maximal length limit . The experimental procedures as well as the choice of architectural design are well explained and designed in a logical way. Substantial comparison experiments also pinpoint the performance. However, it doesn’t compare the result of BERTAL to other text summarizers with arbitrary-length text, which is not based on BERT.