  On a wide range of natural language processing and information retrieval
tasks, transformer-based models, particularly pre-trained language models like
BERT, have demonstrated tremendous effectiveness. Due to the quadratic
complexity of the self-attention mechanism, however, such models have
difficulties processing long documents. Recent works dealing with this issue
include truncating long documents, in which case one loses potential relevant
information, segmenting them into several passages, which may lead to miss some
information and high computational complexity when the number of passages is
large, or modifying the self-attention mechanism to make it sparser as in
sparse-attention models, at the risk again of missing some information. We
follow here a slightly different approach in which one first selects key blocks
of a long document by local query-block pre-ranking, and then few blocks are
aggregated to form a short document that can be processed by a model such as
BERT. Experiments conducted on standard Information Retrieval datasets
demonstrate the effectiveness of the proposed approach.

Information Retrieval