contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on
WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting
long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization
dataset.
Tips:
[LEDForConditionalGeneration] is an extension of
  [BartForConditionalGeneration] exchanging the traditional self-attention layer with
  Longformer's chunked self-attention layer. [LEDTokenizer] is an alias of
  [BartTokenizer].
LED works very well on long-range sequence-to-sequence tasks where the input_ids largely exceed a length of
