Should the option to run on TPU in run_glue.py use some sort of xla data parallelizer ? 
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

The xla API ( https://github.com/pytorch/xla/blob/master/API_GUIDE.md ) and the TPU colab examples ( https://github.com/pytorch/xla/tree/master/contrib/colab ) each parallelize their data, either using a `torch_xla.distributed.parallel_loader.c` object or a `torch_xla.distributed.data_parallel.DataParallel` object (which uses `ParallelLoader`).

The `run_glue.py` example (https://github.com/huggingface/transformers/blob/master/examples/run_glue.py ) doesn't do this. I am wondering if there was a reason not to use xla's data parallelizers.
Indeed, it would be great to improve the current TPU script to include better optimization, such as using the TPU DataParallel from Pytorch. We haven't gotten to it yet and we'll probably do so soon.

We'd be very happy to welcome a PR too! :)<|||||>Sounds good, I'm trying to figure it out. The part I'm stuck on is that the colab TPU examples use a multi-threading approach, but the official API recommends using multi-processing over multi-threading, so I am wonder if the TPUs require a multi-threading approach. 

Once I figure this out I think I can update the code. I asked a question on the xla repo about this
https://github.com/pytorch/xla/issues/1217

<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
