Have you considered using falsh attention to speed up?
### Feature request

using flash attention to speed up

### Motivation

none

### Your contribution

none
cc @younesbelkada <|||||>Hi @macheng6 
Thanks a lot for your interest in this! 
Indeed there is a similar integration to this called `BetterTransformer` that uses flash attention in the backend I believe. 
This support most of the encoder and decoder models (if you use the `main` branch of `optimum`, please refer to this documentation page: https://huggingface.co/docs/optimum/bettertransformer/overview

cc @fxmarty @michaelbenayoun <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.