Is training distilbert with TPU supported yet?
# 🚀 Feature request

Hi! I tried training my own distilbert model with [this code](https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py) using GPU, and it was a success.
I'm wondering if training distilbert model with TPU is supported yet, or if there's any plan to release a new version in which TPU is supported?


Distillation is not supported (yet) on TPU. You can check the status of different example scripts [here](https://github.com/huggingface/transformers/tree/master/examples). Those supported by `Trainer` or `TFTrainer` or `pytorch-lightning` can be run on TPU, others cannot.