There are a few contributed notebooks which might help you here: https://github.com/huggingface/transformers/tree/master/notebooks 628
For instance this one by @valhalla is about training T5 on TPU in PyTorch: https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil 568