finetune_trainer being really slow on TPU
Hi
I need to run finetune_trainer for a large scale project on TPU, here is the command I use 

`python xla_spawn.py --num_cores 8 finetune_trainer.py     --learning_rate=3e-5         --n_val 1000  --model_name_or_path t5-small --data_dir data/wmt_en_ro/ --output_dir /home/rabeeh/temp/ --overwrite_output_dir  --tpu_num_cores=8 --max_source_length=64 --max_target_length=64 --per_device_train_batch_size=32 --per_device_eval_batch_size=32 --label_smoothing=0.1 --task="translation" --logging_steps=200 --eval_steps=500 --num_train_epochs=1 --save_steps=500 --max_source_length=128  --max_target_length=128 --val_max_target_length=128 --test_max_target_length=128
`

It is running slow for me, and I was wondering if I can make it faster, can I make it run distributedly on multiple host machines? I really appreciate providing me with command to run this code efficiently on TPUs on cluster. Is there any setting I am missing in my command? 

thank you. 
could you give me some numbers how much time it could take for training WMT on TPUs so I could compare? I appreciate letting me know how to do the setting on TPUs @patil-suraj <|||||>AFAIK, there is a recent slowdown on TPU that @patil-suraj is investigating. 

I simple speedup would be to use `--model_name_or_path Helsinki-NLP/opus-mt-en-ro`
<|||||>Hi Sam. I need to use T5 and I really got a very slow performance. could you tell me also the batch_size_per_host for v2-8 machine? since I also got SEGFAULT when using batch_size of 32 even per host. I really need to make this work for a project and I really need help. I appreciate assisting me with setting things up on TPU. this is really very slow and I do not know what I can do about it. thanks <|||||>Hi Sam, 
Is there any chance to set a 10 minutes time with me, if I could ask some questions on setting up things on TPUs? I really do not have access to ask anyone questions on this and I need help to set things up. I greatly appreciate it thanks <|||||>could you tell me the type of instance I need to create? maybe I am making mistake with using cloud? thanks<|||||>Sorry for interrupting.

I'm sorry if I'm wrong, but I'm thinking that how about using `--tpu_num_cores` instead of `--num_cores`.

I'm a complete newbie to TPU and I'm currently seeking a way to use of TPU in the Transformer Trainer and I found this issue.
Currently, the `TrainingArguments` class seems to use `tpu_num_cores` as an argument. You may be able to use `tpu_num_cores` instead of `num_cores`.
I'm trying to figure out what `tpu_num_cores` has to do with `num_cores`.

https://github.com/huggingface/transformers/blob/master/src/transformers/training_args.py#L290

I'm sorry if I'm misguided.<|||||>Excuse me, you are using `xla_spawn.py`, and it passes its `--num_cores` as `--tpu_num_cores` to `finetune_trainer.py`.
I apologize for commenting without fully understanding the situation.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
