global_attention_mask at run-time appropriately. All Longformer models employ the following logic for
global_attention_mask:
0: the token attends "locally",
1: the token attends "globally".
For more information please also refer to [~LongformerModel.forward] method.
Using Longformer self attention, the memory and time complexity of the query-key matmul operation, which usually
represents the memory and time bottleneck, can be reduced from \(\mathcal{O}(n_s \times n_s)\) to
\(\mathcal{O}(n_s \times w)\), with \(n_s\) being the sequence length and \(w\) being the average window
size. It is assumed that the number of "globally" attending tokens is insignificant as compared to the number of
"locally" attending tokens.
For more information, please refer to the official paper.
Training
[LongformerForMaskedLM] is trained the exact same way [RobertaForMaskedLM] is
