Add support for `encoder_hidden_states` and `encoder_attention_mask` in modeling_longformer
# 🚀 Feature request

In order to use the longformer with the `EncoderDecoderModel` structure, it needs to accept `head_mask`, `encoder_hidden_states`, and `encoder_attention_mask` in the `forward()` function. Currently, `LongformerSelfAttention` asserts that `encoder_hidden_states` and `encoder_attention_mask` are `None`. I attempted to implement this functionality by comparing the bert implementation, which accepts these values, to the longformer implementation.

I can create an `EncoderDecoderModel` like so: `model = EncoderDecoderModel.from_encoder_decoder_pretrained("allenai/longformer-base-4096", "allenai/longformer-base-4096")` and the model does train using my changes. Even though the model trains, I am not confident that the changes I made are correct. I think the cross attention mechanism already exists within the `LongformerModel` since it is a subclass of `RobertaModel` which is a subclass of `BertModel`. `BertModel` uses `BertLayer`s which use cross attention if `self.is_decoder` is True. 

Furthermore, I get the below error messages, which have something to do with the inheritance hierarchy. It fails to initialize the cross attention layers, which makes sense, but I am not sure about the other ones.

```
2020-06-21 18:00:53,533|transformers.modeling_encoder_decoder|INFO> Initializing allenai/longformer-base-4096 as a decoder model. Cross attention layers are added to allenai/longformer-base-4096 and randomly initialized if allenai/longformer-base-4096's architecture allows for cross attention layers.
2020-06-21 18:00:53,609|transformers.modeling_utils|INFO> loading weights file https://cdn.huggingface.co/allenai/longformer-base-4096/pytorch_model.bin from cache at /root/.cache/torch/transformers/dfc92dbbf5c555abf807425ebdb22b55de7a17e21fe1c48cbaa5764982c1d9c0.cd65234711d2e83d420aa696eb9186cdec6ab79ef8bf090b442cf249443dfa92
2020-06-21 18:00:58,792|transformers.modeling_utils|INFO> Weights of BertLMHeadModel not initialized from pretrained model: ['embeddings.word_embeddings.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.LayerNorm.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.crossattention.self.query.weight', 'encoder.layer.0.crossattention.self.query.bias', 'encoder.layer.0.crossattention.self.key.weight', 'encoder.layer.0.crossattention.self.key.bias', 'encoder.layer.0.crossattention.self.value.weight', 'encoder.layer.0.crossattention.self.value.bias', 'encoder.layer.0.crossattention.output.dense.weight', 'encoder.layer.0.crossattention.output.dense.bias', 'encoder.layer.0.crossattention.output.LayerNorm.weight', 'encoder.layer.0.crossattention.output.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.crossattention.self.query.weight', 'encoder.layer.1.crossattention.self.query.bias', 'encoder.layer.1.crossattention.self.key.weight', 'encoder.layer.1.crossattention.self.key.bias', 'encoder.layer.1.crossattention.self.value.weight', 'encoder.layer.1.crossattention.self.value.bias', 'encoder.layer.1.crossattention.output.dense.weight', 'encoder.layer.1.crossattention.output.dense.bias', 'encoder.layer.1.crossattention.output.LayerNorm.weight', 'encoder.layer.1.crossattention.output.LayerNorm.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.crossattention.self.query.weight', 'encoder.layer.2.crossattention.self.query.bias', 'encoder.layer.2.crossattention.self.key.weight', 'encoder.layer.2.crossattention.self.key.bias', 'encoder.layer.2.crossattention.self.value.weight', 'encoder.layer.2.crossattention.self.value.bias', 'encoder.layer.2.crossattention.output.dense.weight', 'encoder.layer.2.crossattention.output.dense.bias', 'encoder.layer.2.crossattention.output.LayerNorm.weight', 'encoder.layer.2.crossattention.output.LayerNorm.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.crossattention.self.query.weight', 'encoder.layer.3.crossattention.self.query.bias', 'encoder.layer.3.crossattention.self.key.weight', 'encoder.layer.3.crossattention.self.key.bias', 'encoder.layer.3.crossattention.self.value.weight', 'encoder.layer.3.crossattention.self.value.bias', 'encoder.layer.3.crossattention.output.dense.weight', 'encoder.layer.3.crossattention.output.dense.bias', 'encoder.layer.3.crossattention.output.LayerNorm.weight', 'encoder.layer.3.crossattention.output.LayerNorm.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.crossattention.self.query.weight', 'encoder.layer.4.crossattention.self.query.bias', 'encoder.layer.4.crossattention.self.key.weight', 'encoder.layer.4.crossattention.self.key.bias', 'encoder.layer.4.crossattention.self.value.weight', 'encoder.layer.4.crossattention.self.value.bias', 'encoder.layer.4.crossattention.output.dense.weight', 'encoder.layer.4.crossattention.output.dense.bias', 'encoder.layer.4.crossattention.output.LayerNorm.weight', 'encoder.layer.4.crossattention.output.LayerNorm.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.crossattention.self.query.weight', 'encoder.layer.5.crossattention.self.query.bias', 'encoder.layer.5.crossattention.self.key.weight', 'encoder.layer.5.crossattention.self.key.bias', 'encoder.layer.5.crossattention.self.value.weight', 'encoder.layer.5.crossattention.self.value.bias', 'encoder.layer.5.crossattention.output.dense.weight', 'encoder.layer.5.crossattention.output.dense.bias', 'encoder.layer.5.crossattention.output.LayerNorm.weight', 'encoder.layer.5.crossattention.output.LayerNorm.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.crossattention.self.query.weight', 'encoder.layer.6.crossattention.self.query.bias', 'encoder.layer.6.crossattention.self.key.weight', 'encoder.layer.6.crossattention.self.key.bias', 'encoder.layer.6.crossattention.self.value.weight', 'encoder.layer.6.crossattention.self.value.bias', 'encoder.layer.6.crossattention.output.dense.weight', 'encoder.layer.6.crossattention.output.dense.bias', 'encoder.layer.6.crossattention.output.LayerNorm.weight', 'encoder.layer.6.crossattention.output.LayerNorm.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.crossattention.self.query.weight', 'encoder.layer.7.crossattention.self.query.bias', 'encoder.layer.7.crossattention.self.key.weight', 'encoder.layer.7.crossattention.self.key.bias', 'encoder.layer.7.crossattention.self.value.weight', 'encoder.layer.7.crossattention.self.value.bias', 'encoder.layer.7.crossattention.output.dense.weight', 'encoder.layer.7.crossattention.output.dense.bias', 'encoder.layer.7.crossattention.output.LayerNorm.weight', 'encoder.layer.7.crossattention.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.crossattention.self.query.weight', 'encoder.layer.8.crossattention.self.query.bias', 'encoder.layer.8.crossattention.self.key.weight', 'encoder.layer.8.crossattention.self.key.bias', 'encoder.layer.8.crossattention.self.value.weight', 'encoder.layer.8.crossattention.self.value.bias', 'encoder.layer.8.crossattention.output.dense.weight', 'encoder.layer.8.crossattention.output.dense.bias', 'encoder.layer.8.crossattention.output.LayerNorm.weight', 'encoder.layer.8.crossattention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.crossattention.self.query.weight', 'encoder.layer.9.crossattention.self.query.bias', 'encoder.layer.9.crossattention.self.key.weight', 'encoder.layer.9.crossattention.self.key.bias', 'encoder.layer.9.crossattention.self.value.weight', 'encoder.layer.9.crossattention.self.value.bias', 'encoder.layer.9.crossattention.output.dense.weight', 'encoder.layer.9.crossattention.output.dense.bias', 'encoder.layer.9.crossattention.output.LayerNorm.weight', 'encoder.layer.9.crossattention.output.LayerNorm.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.output.dense.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.crossattention.self.query.weight', 'encoder.layer.10.crossattention.self.query.bias', 'encoder.layer.10.crossattention.self.key.weight', 'encoder.layer.10.crossattention.self.key.bias', 'encoder.layer.10.crossattention.self.value.weight', 'encoder.layer.10.crossattention.self.value.bias', 'encoder.layer.10.crossattention.output.dense.weight', 'encoder.layer.10.crossattention.output.dense.bias', 'encoder.layer.10.crossattention.output.LayerNorm.weight', 'encoder.layer.10.crossattention.output.LayerNorm.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.crossattention.self.query.weight', 'encoder.layer.11.crossattention.self.query.bias', 'encoder.layer.11.crossattention.self.key.weight', 'encoder.layer.11.crossattention.self.key.bias', 'encoder.layer.11.crossattention.self.value.weight', 'encoder.layer.11.crossattention.self.value.bias', 'encoder.layer.11.crossattention.output.dense.weight', 'encoder.layer.11.crossattention.output.dense.bias', 'encoder.layer.11.crossattention.output.LayerNorm.weight', 'encoder.layer.11.crossattention.output.LayerNorm.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.LayerNorm.bias', 'pooler.dense.weight', 'pooler.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias']
2020-06-21 18:00:58,816|transformers.modeling_utils|INFO> Weights from pretrained model not used in BertLMHeadModel: ['longformer.embeddings.word_embeddings.weight', 'longformer.embeddings.position_embeddings.weight', 'longformer.embeddings.token_type_embeddings.weight', 'longformer.embeddings.LayerNorm.weight', 'longformer.embeddings.LayerNorm.bias', 'longformer.encoder.layer.0.attention.self.query.weight', 'longformer.encoder.layer.0.attention.self.query.bias', 'longformer.encoder.layer.0.attention.self.key.weight', 'longformer.encoder.layer.0.attention.self.key.bias', 'longformer.encoder.layer.0.attention.self.value.weight', 'longformer.encoder.layer.0.attention.self.value.bias', 'longformer.encoder.layer.0.attention.self.query_global.weight', 'longformer.encoder.layer.0.attention.self.query_global.bias', 'longformer.encoder.layer.0.attention.self.key_global.weight', 'longformer.encoder.layer.0.attention.self.key_global.bias', 'longformer.encoder.layer.0.attention.self.value_global.weight', 'longformer.encoder.layer.0.attention.self.value_global.bias', 'longformer.encoder.layer.0.attention.output.dense.weight', 'longformer.encoder.layer.0.attention.output.dense.bias', 'longformer.encoder.layer.0.attention.output.LayerNorm.weight', 'longformer.encoder.layer.0.attention.output.LayerNorm.bias', 'longformer.encoder.layer.0.intermediate.dense.weight', 'longformer.encoder.layer.0.intermediate.dense.bias', 'longformer.encoder.layer.0.output.dense.weight', 'longformer.encoder.layer.0.output.dense.bias', 'longformer.encoder.layer.0.output.LayerNorm.weight', 'longformer.encoder.layer.0.output.LayerNorm.bias', 'longformer.encoder.layer.1.attention.self.query.weight', 'longformer.encoder.layer.1.attention.self.query.bias', 'longformer.encoder.layer.1.attention.self.key.weight', 'longformer.encoder.layer.1.attention.self.key.bias', 'longformer.encoder.layer.1.attention.self.value.weight', 'longformer.encoder.layer.1.attention.self.value.bias', 'longformer.encoder.layer.1.attention.self.query_global.weight', 'longformer.encoder.layer.1.attention.self.query_global.bias', 'longformer.encoder.layer.1.attention.self.key_global.weight', 'longformer.encoder.layer.1.attention.self.key_global.bias', 'longformer.encoder.layer.1.attention.self.value_global.weight', 'longformer.encoder.layer.1.attention.self.value_global.bias', 'longformer.encoder.layer.1.attention.output.dense.weight', 'longformer.encoder.layer.1.attention.output.dense.bias', 'longformer.encoder.layer.1.attention.output.LayerNorm.weight', 'longformer.encoder.layer.1.attention.output.LayerNorm.bias', 'longformer.encoder.layer.1.intermediate.dense.weight', 'longformer.encoder.layer.1.intermediate.dense.bias', 'longformer.encoder.layer.1.output.dense.weight', 'longformer.encoder.layer.1.output.dense.bias', 'longformer.encoder.layer.1.output.LayerNorm.weight', 'longformer.encoder.layer.1.output.LayerNorm.bias', 'longformer.encoder.layer.2.attention.self.query.weight', 'longformer.encoder.layer.2.attention.self.query.bias', 'longformer.encoder.layer.2.attention.self.key.weight', 'longformer.encoder.layer.2.attention.self.key.bias', 'longformer.encoder.layer.2.attention.self.value.weight', 'longformer.encoder.layer.2.attention.self.value.bias', 'longformer.encoder.layer.2.attention.self.query_global.weight', 'longformer.encoder.layer.2.attention.self.query_global.bias', 'longformer.encoder.layer.2.attention.self.key_global.weight', 'longformer.encoder.layer.2.attention.self.key_global.bias', 'longformer.encoder.layer.2.attention.self.value_global.weight', 'longformer.encoder.layer.2.attention.self.value_global.bias', 'longformer.encoder.layer.2.attention.output.dense.weight', 'longformer.encoder.layer.2.attention.output.dense.bias', 'longformer.encoder.layer.2.attention.output.LayerNorm.weight', 'longformer.encoder.layer.2.attention.output.LayerNorm.bias', 'longformer.encoder.layer.2.intermediate.dense.weight', 'longformer.encoder.layer.2.intermediate.dense.bias', 'longformer.encoder.layer.2.output.dense.weight', 'longformer.encoder.layer.2.output.dense.bias', 'longformer.encoder.layer.2.output.LayerNorm.weight', 'longformer.encoder.layer.2.output.LayerNorm.bias', 'longformer.encoder.layer.3.attention.self.query.weight', 'longformer.encoder.layer.3.attention.self.query.bias', 'longformer.encoder.layer.3.attention.self.key.weight', 'longformer.encoder.layer.3.attention.self.key.bias', 'longformer.encoder.layer.3.attention.self.value.weight', 'longformer.encoder.layer.3.attention.self.value.bias', 'longformer.encoder.layer.3.attention.self.query_global.weight', 'longformer.encoder.layer.3.attention.self.query_global.bias', 'longformer.encoder.layer.3.attention.self.key_global.weight', 'longformer.encoder.layer.3.attention.self.key_global.bias', 'longformer.encoder.layer.3.attention.self.value_global.weight', 'longformer.encoder.layer.3.attention.self.value_global.bias', 'longformer.encoder.layer.3.attention.output.dense.weight', 'longformer.encoder.layer.3.attention.output.dense.bias', 'longformer.encoder.layer.3.attention.output.LayerNorm.weight', 'longformer.encoder.layer.3.attention.output.LayerNorm.bias', 'longformer.encoder.layer.3.intermediate.dense.weight', 'longformer.encoder.layer.3.intermediate.dense.bias', 'longformer.encoder.layer.3.output.dense.weight', 'longformer.encoder.layer.3.output.dense.bias', 'longformer.encoder.layer.3.output.LayerNorm.weight', 'longformer.encoder.layer.3.output.LayerNorm.bias', 'longformer.encoder.layer.4.attention.self.query.weight', 'longformer.encoder.layer.4.attention.self.query.bias', 'longformer.encoder.layer.4.attention.self.key.weight', 'longformer.encoder.layer.4.attention.self.key.bias', 'longformer.encoder.layer.4.attention.self.value.weight', 'longformer.encoder.layer.4.attention.self.value.bias', 'longformer.encoder.layer.4.attention.self.query_global.weight', 'longformer.encoder.layer.4.attention.self.query_global.bias', 'longformer.encoder.layer.4.attention.self.key_global.weight', 'longformer.encoder.layer.4.attention.self.key_global.bias', 'longformer.encoder.layer.4.attention.self.value_global.weight', 'longformer.encoder.layer.4.attention.self.value_global.bias', 'longformer.encoder.layer.4.attention.output.dense.weight', 'longformer.encoder.layer.4.attention.output.dense.bias', 'longformer.encoder.layer.4.attention.output.LayerNorm.weight', 'longformer.encoder.layer.4.attention.output.LayerNorm.bias', 'longformer.encoder.layer.4.intermediate.dense.weight', 'longformer.encoder.layer.4.intermediate.dense.bias', 'longformer.encoder.layer.4.output.dense.weight', 'longformer.encoder.layer.4.output.dense.bias', 'longformer.encoder.layer.4.output.LayerNorm.weight', 'longformer.encoder.layer.4.output.LayerNorm.bias', 'longformer.encoder.layer.5.attention.self.query.weight', 'longformer.encoder.layer.5.attention.self.query.bias', 'longformer.encoder.layer.5.attention.self.key.weight', 'longformer.encoder.layer.5.attention.self.key.bias', 'longformer.encoder.layer.5.attention.self.value.weight', 'longformer.encoder.layer.5.attention.self.value.bias', 'longformer.encoder.layer.5.attention.self.query_global.weight', 'longformer.encoder.layer.5.attention.self.query_global.bias', 'longformer.encoder.layer.5.attention.self.key_global.weight', 'longformer.encoder.layer.5.attention.self.key_global.bias', 'longformer.encoder.layer.5.attention.self.value_global.weight', 'longformer.encoder.layer.5.attention.self.value_global.bias', 'longformer.encoder.layer.5.attention.output.dense.weight', 'longformer.encoder.layer.5.attention.output.dense.bias', 'longformer.encoder.layer.5.attention.output.LayerNorm.weight', 'longformer.encoder.layer.5.attention.output.LayerNorm.bias', 'longformer.encoder.layer.5.intermediate.dense.weight', 'longformer.encoder.layer.5.intermediate.dense.bias', 'longformer.encoder.layer.5.output.dense.weight', 'longformer.encoder.layer.5.output.dense.bias', 'longformer.encoder.layer.5.output.LayerNorm.weight', 'longformer.encoder.layer.5.output.LayerNorm.bias', 'longformer.encoder.layer.6.attention.self.query.weight', 'longformer.encoder.layer.6.attention.self.query.bias', 'longformer.encoder.layer.6.attention.self.key.weight', 'longformer.encoder.layer.6.attention.self.key.bias', 'longformer.encoder.layer.6.attention.self.value.weight', 'longformer.encoder.layer.6.attention.self.value.bias', 'longformer.encoder.layer.6.attention.self.query_global.weight', 'longformer.encoder.layer.6.attention.self.query_global.bias', 'longformer.encoder.layer.6.attention.self.key_global.weight', 'longformer.encoder.layer.6.attention.self.key_global.bias', 'longformer.encoder.layer.6.attention.self.value_global.weight', 'longformer.encoder.layer.6.attention.self.value_global.bias', 'longformer.encoder.layer.6.attention.output.dense.weight', 'longformer.encoder.layer.6.attention.output.dense.bias', 'longformer.encoder.layer.6.attention.output.LayerNorm.weight', 'longformer.encoder.layer.6.attention.output.LayerNorm.bias', 'longformer.encoder.layer.6.intermediate.dense.weight', 'longformer.encoder.layer.6.intermediate.dense.bias', 'longformer.encoder.layer.6.output.dense.weight', 'longformer.encoder.layer.6.output.dense.bias', 'longformer.encoder.layer.6.output.LayerNorm.weight', 'longformer.encoder.layer.6.output.LayerNorm.bias', 'longformer.encoder.layer.7.attention.self.query.weight', 'longformer.encoder.layer.7.attention.self.query.bias', 'longformer.encoder.layer.7.attention.self.key.weight', 'longformer.encoder.layer.7.attention.self.key.bias', 'longformer.encoder.layer.7.attention.self.value.weight', 'longformer.encoder.layer.7.attention.self.value.bias', 'longformer.encoder.layer.7.attention.self.query_global.weight', 'longformer.encoder.layer.7.attention.self.query_global.bias', 'longformer.encoder.layer.7.attention.self.key_global.weight', 'longformer.encoder.layer.7.attention.self.key_global.bias', 'longformer.encoder.layer.7.attention.self.value_global.weight', 'longformer.encoder.layer.7.attention.self.value_global.bias', 'longformer.encoder.layer.7.attention.output.dense.weight', 'longformer.encoder.layer.7.attention.output.dense.bias', 'longformer.encoder.layer.7.attention.output.LayerNorm.weight', 'longformer.encoder.layer.7.attention.output.LayerNorm.bias', 'longformer.encoder.layer.7.intermediate.dense.weight', 'longformer.encoder.layer.7.intermediate.dense.bias', 'longformer.encoder.layer.7.output.dense.weight', 'longformer.encoder.layer.7.output.dense.bias', 'longformer.encoder.layer.7.output.LayerNorm.weight', 'longformer.encoder.layer.7.output.LayerNorm.bias', 'longformer.encoder.layer.8.attention.self.query.weight', 'longformer.encoder.layer.8.attention.self.query.bias', 'longformer.encoder.layer.8.attention.self.key.weight', 'longformer.encoder.layer.8.attention.self.key.bias', 'longformer.encoder.layer.8.attention.self.value.weight', 'longformer.encoder.layer.8.attention.self.value.bias', 'longformer.encoder.layer.8.attention.self.query_global.weight', 'longformer.encoder.layer.8.attention.self.query_global.bias', 'longformer.encoder.layer.8.attention.self.key_global.weight', 'longformer.encoder.layer.8.attention.self.key_global.bias', 'longformer.encoder.layer.8.attention.self.value_global.weight', 'longformer.encoder.layer.8.attention.self.value_global.bias', 'longformer.encoder.layer.8.attention.output.dense.weight', 'longformer.encoder.layer.8.attention.output.dense.bias', 'longformer.encoder.layer.8.attention.output.LayerNorm.weight', 'longformer.encoder.layer.8.attention.output.LayerNorm.bias', 'longformer.encoder.layer.8.intermediate.dense.weight', 'longformer.encoder.layer.8.intermediate.dense.bias', 'longformer.encoder.layer.8.output.dense.weight', 'longformer.encoder.layer.8.output.dense.bias', 'longformer.encoder.layer.8.output.LayerNorm.weight', 'longformer.encoder.layer.8.output.LayerNorm.bias', 'longformer.encoder.layer.9.attention.self.query.weight', 'longformer.encoder.layer.9.attention.self.query.bias', 'longformer.encoder.layer.9.attention.self.key.weight', 'longformer.encoder.layer.9.attention.self.key.bias', 'longformer.encoder.layer.9.attention.self.value.weight', 'longformer.encoder.layer.9.attention.self.value.bias', 'longformer.encoder.layer.9.attention.self.query_global.weight', 'longformer.encoder.layer.9.attention.self.query_global.bias', 'longformer.encoder.layer.9.attention.self.key_global.weight', 'longformer.encoder.layer.9.attention.self.key_global.bias', 'longformer.encoder.layer.9.attention.self.value_global.weight', 'longformer.encoder.layer.9.attention.self.value_global.bias', 'longformer.encoder.layer.9.attention.output.dense.weight', 'longformer.encoder.layer.9.attention.output.dense.bias', 'longformer.encoder.layer.9.attention.output.LayerNorm.weight', 'longformer.encoder.layer.9.attention.output.LayerNorm.bias', 'longformer.encoder.layer.9.intermediate.dense.weight', 'longformer.encoder.layer.9.intermediate.dense.bias', 'longformer.encoder.layer.9.output.dense.weight', 'longformer.encoder.layer.9.output.dense.bias', 'longformer.encoder.layer.9.output.LayerNorm.weight', 'longformer.encoder.layer.9.output.LayerNorm.bias', 'longformer.encoder.layer.10.attention.self.query.weight', 'longformer.encoder.layer.10.attention.self.query.bias', 'longformer.encoder.layer.10.attention.self.key.weight', 'longformer.encoder.layer.10.attention.self.key.bias', 'longformer.encoder.layer.10.attention.self.value.weight', 'longformer.encoder.layer.10.attention.self.value.bias', 'longformer.encoder.layer.10.attention.self.query_global.weight', 'longformer.encoder.layer.10.attention.self.query_global.bias', 'longformer.encoder.layer.10.attention.self.key_global.weight', 'longformer.encoder.layer.10.attention.self.key_global.bias', 'longformer.encoder.layer.10.attention.self.value_global.weight', 'longformer.encoder.layer.10.attention.self.value_global.bias', 'longformer.encoder.layer.10.attention.output.dense.weight', 'longformer.encoder.layer.10.attention.output.dense.bias', 'longformer.encoder.layer.10.attention.output.LayerNorm.weight', 'longformer.encoder.layer.10.attention.output.LayerNorm.bias', 'longformer.encoder.layer.10.intermediate.dense.weight', 'longformer.encoder.layer.10.intermediate.dense.bias', 'longformer.encoder.layer.10.output.dense.weight', 'longformer.encoder.layer.10.output.dense.bias', 'longformer.encoder.layer.10.output.LayerNorm.weight', 'longformer.encoder.layer.10.output.LayerNorm.bias', 'longformer.encoder.layer.11.attention.self.query.weight', 'longformer.encoder.layer.11.attention.self.query.bias', 'longformer.encoder.layer.11.attention.self.key.weight', 'longformer.encoder.layer.11.attention.self.key.bias', 'longformer.encoder.layer.11.attention.self.value.weight', 'longformer.encoder.layer.11.attention.self.value.bias', 'longformer.encoder.layer.11.attention.self.query_global.weight', 'longformer.encoder.layer.11.attention.self.query_global.bias', 'longformer.encoder.layer.11.attention.self.key_global.weight', 'longformer.encoder.layer.11.attention.self.key_global.bias', 'longformer.encoder.layer.11.attention.self.value_global.weight', 'longformer.encoder.layer.11.attention.self.value_global.bias', 'longformer.encoder.layer.11.attention.output.dense.weight', 'longformer.encoder.layer.11.attention.output.dense.bias', 'longformer.encoder.layer.11.attention.output.LayerNorm.weight', 'longformer.encoder.layer.11.attention.output.LayerNorm.bias', 'longformer.encoder.layer.11.intermediate.dense.weight', 'longformer.encoder.layer.11.intermediate.dense.bias', 'longformer.encoder.layer.11.output.dense.weight', 'longformer.encoder.layer.11.output.dense.bias', 'longformer.encoder.layer.11.output.LayerNorm.weight', 'longformer.encoder.layer.11.output.LayerNorm.bias', 'longformer.pooler.dense.weight', 'longformer.pooler.dense.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']
2020-06-21 18:00:58,818|transformers.configuration_encoder_decoder|INFO> Set `config.is_decoder=True` for decoder_config
```

## Motivation

I'd like to use the longformer as an `EncoderDecoderModel` and these changes are required for the `EncoderDecoderModel` to work properly. My end goal is to use a encoder-decoder architecture for long documents. The better solution appears to be #4406. However, I still wonder if my changes are correct.

## Your contribution

My changes are in the `longformer-encoder_hidden_states` branch of [HHousen/transformers](https://github.com/HHousen/transformers/tree/longformer-encoder_hidden_states). Link to modified file: [modeling_longformer.py](https://github.com/HHousen/transformers/blob/longformer-encoder_hidden_states/src/transformers/modeling_longformer.py).


Hey @HHousen,

In order for Longformer to work some changes need to be done to the `longformer_modeling.py` file (move the level of abstraction much lower) and also we need to think about how to deal with cross - attention layers for Longformer's chunked self attention. I will do some refactoring for Longformer soon. 

As it is now, I will not work properly within the encoder-decoder architecture<|||||>Linking this to https://github.com/huggingface/transformers/issues/4225#issuecomment-659467998.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
