Using high-level keras to fine-tune T5 seems tricky due to limited flexibility and code controls. I wrote something on fine-tuning T5 with customized training loop in TF2.0: https://github.com/wangcongcong123/ttt 69. Hope this helps.