A minor inconsistency: on a GPU runtime, when I execute:
!pip install -q cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.9-cp37-cp37m-linux_x86_64.whl

to install the TPU client, HuggingFace will try to use XLA even if a CUDA device is present.
RuntimeError: tensorflow/compiler/xla/xla_client/computation_client.cc:273 : Missing XLA configuration

Shouldn’t there be checks to verify if XLA/TPU cores flag is not passed, it should fall back to CUDA->CPU rather than trying to run via XLA?