Optimize by not computing gradients for parameters set to requires_grad=False
Fix  #21182 

@sgugger 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21236). All of your documentation changes will be reflected on that endpoint.<|||||>@sgugger  Need to retrigger this build .
