Where is the impact when output_attentions=True?
Is there any impact regarding performance (training/fine-tuning time, GPU memory, batch size, etc.) when  `output_attentions=True`?

```python
self.bert_encoder = BertModel.from_pretrained(
            hparams.architecture, # "bert-base-uncased"
            output_attentions=True)
```
If `output_attentions=True` memory consumption should increase significantly (for large `sequence_length`) since we now store all attentions of size (`batch_size`, `num_heas`, `sequence_length`, `sequence_length`). This is less significant in training since the stored activations for training consume most RAM anyways. Speed should not be really affected by this.<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.