evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In
contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on
WikiHop and TriviaQA.
Tips:
Since the Longformer is based on RoBERTa, it doesn't have token_type_ids. You don't need to indicate which
  token belongs to which segment. Just separate your segments with the separation token tokenizer.sep_token (or
  </s>).
This model was contributed by beltagy. The Authors' code can be found here.
Longformer Self Attention
Longformer self attention employs self attention on both a "local" context and a "global" context. Most tokens only
