Thanks very much Patrick @patrickvonplaten!
In Suraj Patil’s notebook, he employed Pytorch Trainer to train T5.
At first, I didn’t know that we can use Trainer with Seq2Seq problems (according to “The Big Table of Tasks”  https://huggingface.co/transformers/examples.html 30 which stated that Trainer does not yet support Translation / Summarization )
I will try to use TFTrainer for TF2 on Seq2Seq problems. If that doesn’t work, I think I will try to write custom loop in TF2.