Better TPU Support in examples
# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

I tried to train BERT on TPU recently, and found your [examples](https://github.com/huggingface/transformers/blob/master/examples) has done work on this topic. However, some code looks experimental and not perfectly ready to be used. The combination of [xla and pytorch-lightening](https://github.com/huggingface/transformers/blob/master/examples/lightning_base.py) looks great, but seems not be used neither in any training scripts nor in documents. I'd like to know when will those codes be ready, and I'd be very glad to contribute some code.

I agree with this request. TPU training pipeline is very fragile, lacking and needs more attention. Encouraging more use of TPU by providing easy examples would increase its usage resulting in a higher quality TPU Training system over time as more people contribute.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
