Using `bf16` instead of `fp16`
# 🚀 Feature request

As seen in [this pr](https://github.com/huggingface/transformers/pull/10956), there is demand for `bf16` compatibility in training of transformers models.  The pytorch folks just [added this feature](https://github.com/pytorch/pytorch/pull/61002) to their master branch, so we are now able to work on adding it to this repo.

## Motivation

Related to [this issue](https://github.com/huggingface/transformers/pull/10956) and [this pytorch pr](https://github.com/pytorch/pytorch/pull/61002).  This feature would allow for proper half-precision training of google-trained models, for example any `T5` model.

## Your contribution

I am currently working on a PR for this [here](https://github.com/JamesDeAntonis/transformers/tree/bf16), and would gladly field any suggestions and contributions.

@stas00 
Very much looking forward to enable bf16 in PyTorch :-) Think we should probably wait though until the next PyTorch release is out. But it would be a good idea to have it supported as soon as the release is out then. cc @LysandreJik @sgugger <|||||>Indeed. I will install pt-nightly and work on this.

Thank you for staying on top of the pytorch development, @JamesDeAntonis <|||||>@stas00 I'm going to open a pr shortly for my [branch](https://github.com/JamesDeAntonis/transformers/tree/bf16).  Feel free to check that out for starter