Longformer
Longformer: The Long-Document Transformer, Iz Beltagy et al.
A transformer model replacing the attention matrices by sparse matrices to go faster. Often, the local context (e.g.,
what are the two tokens left and right?) is enough to take action for a given token. Some preselected input tokens are
still given global attention, but the attention matrix has way less parameters, resulting in a speed-up. See the
local attention section for more information.
It is pretrained the same way a RoBERTa otherwise.
This model could be very well be used in an autoregressive setting, there is no checkpoint for such a
pretraining yet, though.
The library provides a version of the model for masked language modeling, token classification, sentence
classification, multiple choice classification and question answering.
