trainer.py does not handle distributed training for iterative datasets and is very slow
## Environment info
- `transformers` version: 3.5.1
- Platform: TPU
- Python version: 3.7
- using xla_spawn.py 

### Who can help
@sgugger
@patrickvonplaten
@patrickvonplaten
@patil-suraj

## Information
I am running seq2seq_finetune.py with iterative datasets and I do not get any speed up for 8 TPU cores versus 1 TPU cores, the code is also even slower than 1 GPU.

## To reproduce
```
git clone git@github.com:google-research/ruse.git
go to iter branch 
pip install -r requirements.txt
python setup.py develop
cd seq2seq
python xla_spawn.py finetune_t5_trainer.py  configs/mrpc_adapter_tpu.json

```

## Expected behavior
Being faster on TPU, to me trainer.py does not handle iterative datasets properly. could you have a look please? thank you for your help. 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.