LongFormerAttention For AutoRegressive Models
Longformer Currently supports only Bidirectional Attention,It would be a great feature to finetune current language models like GPT-2 on longer sequences .
Yes! This will be added when starting the `Longformer` Encoder framework :-) Closing this in favor of https://github.com/huggingface/transformers/issues/5170 and https://github.com/huggingface/transformers/issues/4225