-1e9 constants in T5 implementation
These -1e9 constants are too large for fp16 training

https://github.com/huggingface/transformers/blob/df735d1317994e366ab0edff6c55930e18912b7c/src/transformers/models/t5/modeling_tf_t5.py#L728
https://github.com/huggingface/transformers/blob/df735d1317994e366ab0edff6c55930e18912b7c/src/transformers/models/t5/modeling_tf_t5.py#L746

Maybe they should be made configurable?
Probably of interest to @ydshieh 