run_mlm is not working with TPU 
### System Info

I am using colab : 
python:3.10

for torch and xla : 
`pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl`

transformers == 4.30.2


after using this command the training get this error : 

`2023-07-12 21:07:17.770888: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:09.577189: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:09.645535: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:09.848706: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.028873: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.122547: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.322647: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.612495: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.867921: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Exception in device=TPU:0: Invalid device format: cpu
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 334, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 328, in _start_fn
    fn(gindex, *args)
  File "/content/run_mlm.py", line 654, in _mp_fn
    main()
  File "/content/run_mlm.py", line 239, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/usr/local/lib/python3.10/dist-packages/transformers/hf_argparser.py", line 346, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 111, in __init__
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 1341, in __post_init__
    and (get_xla_device_type(self.device) != "GPU")
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 127, in get_xla_device_type
    return xm.xla_real_devices([device])[0].split(":")[0]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 271, in xla_real_devices
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 271, in <listcomp>
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 266, in _xla_real_device
    raise RuntimeError('Invalid device format: {}'.format(device_str))
RuntimeError: Invalid device format: cpu
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/xla_spawn.py:83 in <module>                                         │
│                                                                              │
│   80                                                                         │
│   81                                                                         │
│   82 if __name__ == "__main__":                                              │
│ ❱ 83 │   main()                                                              │
│   84                                                                         │
│                                                                              │
│ /content/xla_spawn.py:79 in main                                             │
│                                                                              │
│   76 │   # Patch sys.argv                                                    │
│   77 │   sys.argv = [args.training_script] + args.training_script_args + ["- │
│   78 │                                                                       │
│ ❱ 79 │   xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)               │
│   80                                                                         │
│   81                                                                         │
│   82 if __name__ == "__main__":                                              │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/torch_xla/distributed/xla_multiproce │
│ ssing.py:397 in spawn                                                        │
│                                                                              │
│   394   if pf_cfg.num_devices == 1:                                          │
│   395 │   _start_fn(0, pf_cfg, fn, args)                                     │
│   396   else:                                                                │
│ ❱ 397 │   result = torch.multiprocessing.start_processes(                    │
│   398 │   │   _mp_start_fn,                                                  │
│   399 │   │   args=(pf_cfg, fn, args),                                       │
│   400 │   │   nprocs=pf_cfg.num_devices,                                     │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py:197   │
│ in start_processes                                                           │
│                                                                              │
│   194 │   │   return context                                                 │
│   195 │                                                                      │
│   196 │   # Loop on join until it returns True or raises an exception.       │
│ ❱ 197 │   while not context.join():                                          │
│   198 │   │   pass                                                           │
│   199                                                                        │
│   200                                                                        │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py:149   │
│ in join                                                                      │
│                                                                              │
│   146 │   │   │   │   │   signal_name=name                                   │
│   147 │   │   │   │   )                                                      │
│   148 │   │   │   else:                                                      │
│ ❱ 149 │   │   │   │   raise ProcessExitedException(                          │
│   150 │   │   │   │   │   "process %d terminated with exit code %d" %        │
│   151 │   │   │   │   │   (error_index, exitcode),                           │
│   152 │   │   │   │   │   error_index=error_index,                           │
╰──────────────────────────────────────────────────────────────────────────────╯
ProcessExitedException: process 0 terminated with exit code 17`


### Who can help?

@sgugger ,  @ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. choose TPU platform on colab 
2. ```
!python xla_spawn.py --num_cores 8 run_mlm.py \
    --model_name_or_path roberta-base \
    --tpu_num_cores 8 \
    --train_file tr.txt \
    --validation_file dev.txt \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --max_seq_len 200 \
    --line_by_line True \
    --pad_to_max_length True \
    --output_dir mlm_tpu-v2
```


### Expected behavior

The training should run.
It looks like PyTorch XLA cannot see your TPUs, are you sure you properly set up your instance?<|||||>I suppose it is, I am using colab TPU and I have installed thiis package : 
`pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl`