ALIBI position embedding support for other models ( BERT, ELECTRA )
### Feature request

Since BLOOM is using ALIBI for position embedding, is there any plans to add ALIBI to other existing models such as BERT, ELECTRA etc?

ALIBI have so far proven to be [extrapolate sequence length beyond training sequence](https://arxiv.org/pdf/2108.12409.pdf) (figure 4). I think this idea should be added to existing architecture since existing models such as BERT, ELECTRA already supports more than one types of position embedding strategy.

### Motivation

If existing models such as BERT, Roberta or ELECTRA supports ALIBI position strategy, this will motivate community pretrained BERT which supports beyond 512 sequence length. One of the largest impact would be long form input sequence in summarization task.

### Your contribution

I have implemented ALIBI mechanism on my own private transformer fork and could submit a PR if this idea is accepted.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>@theblackcat102 I would be quite interested in that. Could you also link your fork?

I would be quite interested to train BERT et al for longer sequences. 