Add a new gradient regularization feature
# 🚀 Feature request

When we fine-tune a large-scale pre-trained model in the downstream task (especially with low-resource data), it would be helpful if we randomly discard some gradients, as introduced in this [paper](https://aclanthology.org/2021.emnlp-main.749.pdf).
Therefore, I wonder if I could add this simple feature to transformers?

## Motivation

Add a simple regularization feature to address the mismatch between large models and small training data.

## Your contribution

I implement the feature by adding the following codes in the `step()` function in the optimizer.

```
grad_mask = Bernoulli(grad.new_full(size=grad.size(), fill_value=self.reserve_p))  
grad *= grad_mask.sample() / self.reserve_p
```

Maybe of interest to @sgugger or @Rocketknight1 :)<|||||>It seems like an interesting paper, but there are **a lot** of papers with new training recipes like this, and I don't think we can implement them all until they become very famous and widely-used. Also, this particular method should be relatively easy for users to implement in their custom training loops or TF `train_step` code, so I think we probably would not be able to accept this PR right now (although we definitely appreciate the enthusiasm!). @sgugger do you agree?<|||||> I agree this is out of scope of the `Trainer` for now, unless it becomes a widely used technique. In the meantime, using our [`Accelerate`](https://github.com/huggingface/accelerate) library lets the user write their custom training loop where they can implement this feature easily :-)<|||||>Thanks so much :) 