Align attention_mask dtype to the attention_scores for BERT to speedup mixed precision inference and training
# Why we need to align the attention_mask dtype to the attention_scores
For the encoder layer in BERT model, the attention_scores need to add the attention_mask info before the add and softmax layer to process the masked batch inputs.
The  attention_mask is always FP32 in this repository, It works well for fully FP32 data type, but when we use mixed precision, the attention_scores is low precision, such as bf16,  and the output of add operation will be FP32 in PyTorch. so the following softmax and dropout will also run with FP32 and introduce addition "to" operation. So, it will obviously reduce the performance for mixed precision performance with FP32 attention_mask. 
# Optimization 
Convert the attention_mask to the same type of  attention_scores  in the first encoder layer. With this optimization, we only need to convert the data type once and the following softmax and dropout will also use low precision data type to reduce memory footprint to improve the performance. 
Thank you for the PR!

Note that `attention_mask` is actually in whatever the given `dtype` is. For bert like models the mask is prepared using `get_extended_attention_mask` method and checks and casts the mask to correct `dtype`. cf
https://github.com/huggingface/transformers/blob/62832c962f85b5a554ebf8b930d13b76b9028a8d/src/transformers/modeling_utils.py#L232

So the attention operation should also be in mixed precision.<|||||>But for modeling_bert.py the attention_mask is generate with 
`extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)`
the dtype of extended_attention_mask is same to attention_mask. while attention_mask is the input of model, it is FP32 in general. For mixed precision in PyTorch, we use autocast to do mixed precision, user doesn't need to convert the input dtype explicitly. So, the attention mask will always be FP32 here.  In act, the dtype of mask should be same to the attention_scores. <|||||>>But for modeling_bert.py the attention_mask is generate with
extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
the dtype of extended_attention_mask is same to attention_mask

if you go through, `get_extended_attention_mask` you'll find that the mask is casted to `self.dtype`, which calls ` get_parameter_dtype`

https://github.com/huggingface/transformers/blob/62832c962f85b5a554ebf8b930d13b76b9028a8d/src/transformers/modeling_utils.py#L130

which returns the `dtype` of parameter, so the `extended_attention_mask` will have the same `dtype` as that of model parameters. So if params are in `fp16` then `extended_attention_mask` will also be in `fp16`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.