TPU slow finetuning T5-base
Hi,
I am trying to train a T5-base on Colab with the TPU. I am using the official code to perform a fine-tuning on the T5-base (with my dataset), but the training with TPU is extremely slow! I'm using the offical [code](https://github.com/huggingface/transformers/blob/master/examples/pytorch/translation/run_translation.py).

I am attaching the colab code with the various libraries I have installed: [notebook](https://colab.research.google.com/drive/1g4PYEOa0zhJnMaThPZVSQepn_wr9md4f#scrollTo=XNVao-sd4fbH).

Also, if I try to increase the batch size as >= 64, I get a memory error, as there seems to be only about 8 Gb available.

Can someone help me? Thank you!
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>@LysandreJik Done days ago, but no answer yet<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.