[Question] Add extra sublayer for each layer of Transformer
## ❓ Questions & Help

Hello !

BERT base has 12 layers and each layer includes follwing sublayer ( and of course, add norm)

` {self-attention -> feed-foward} `

I was wondering if there is a way of adding extra unit to this sublayer , for example,

`{self-attention -> feed-foward -> **LSTM**}`

Thanks
 
Anything is possible, if you want to! But it's not straightforward, I think. You can have a look at `BertLayer` where I would assume that you make your changes.

https://github.com/huggingface/transformers/blob/dfe012ad9d6b6f0c9d30bc508b9f1e4c42280c07/src/transformers/modeling_bert.py#L365-L373<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
