There was an announcement in the PyTorch Developer Conference this week, where a ResNet50 was successfully trained on a TPU.
For more info, check https://cloud.google.com/blog/products/ai-machine-learning/introducing-pytorch-across-google-cloud 852