support Mixed Precision and avoid 'dtype=float32' in implementation
# 🚀 Feature request
When using Longformer model, I found the dtype of many tensors are assigned directly by `dtype=tf.dtypes.float32` or equations, which makes mixed precision raining impossible. And I found that other models also have this problem.

Of course It is not a bug to not support mixed precision training. But because Transformer models are usually very large, it will be appreciated to support mix precision training when implementing new models. 

So I suggest if possible please use dtype inference instead of direct dtype assignment.

Hello!
Thanks for the feature requests. We are currently working on this, some of them already support mixed precision 👍 <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.