Feature Request
I read the article “SesameBERT: Attention for Anywhere 2” and would like to add SENet blocks in the Huggingface implementation. The article’s authors made an implementation with Tensorflow, but I would like to use the lib in pytorch.

Motivation
The use of SENet Blocks has obtained state-of-the-art results. And they seem to be promising in NLP.

Your contribution
I know that it is possible to modify the [BertLayer() and [BertEnconder()] classes
Any suggestions on how to modify the code so that you can apply the idea used in the article?

Captura de Tela 2021-06-02 às 22.50.351060×572 94.3 KB