@sshleifer maybe a simpler question is: has there been a successful effort to fine-tune T5 models with HF?
Btw, I am using version  3.3.1  of transformers, in case there have been any major bug fixes since then.