In case it’s of interest, I’ve uploaded a large mT5 model fine-tuned on MNLI and xtreme-XNLI to the model hub: alan-turing-institute/mt5-large-finetuned-mnli-xtreme-xnli 27
I ended up tuning using the original google repo (with some nice pointers from Stephen Mayhew’s notebook 29). So can’t really offer much in the way of tips for tuning with transformers, unfortunately.
I’ve seen some fairly encouraging early results comparing the tuned model to joddav’s excellent XLM-R model (have run out of links in this post for new user) for zero shot classification on some benchmark data (yinwenpeng, BenchmarkingZeroShot - link limit again!) but have only run over a small subset so far.  My institute is planning on trialing these models in a project we’ve got coming up so hopefully will be able to update at some point