Attention gradients for models
### Feature request

Hi, I was wondering if there is a way to get the gradients of the attention weights. In PyTorch, we can do this via hooks and it works perfectly for getting embeddings gradients. But, I had an issue doing this for the transformer attention weights. Is there any way we can make this possible?

### Motivation

This can help with model interpretability with the scaled attention method.

### Your contribution

I can attach my current codebase if you guys would be interested. It works partly but sometimes the gradients are zeroed. I am not sure if this is the correct behavior though.
Hi @Rachneet ,

Model interpretability is indeed interesting and useful!

However, I think currently we don't have a plan to integrate the mechanism of getting gradients into `transformers`. There is an library [Captum](https://captum.ai/) which might be useful in this area though.

cc @LysandreJik <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.