Trn1 LoRA finetuning with HF reaches RuntimeError: Invalid device format: cpu
### System Info

optimum 1.9.0
optimum-neuron 0.0.7
transformers 4.30.2
DLAMI: https://aws.amazon.com/marketplace/pp/prodview-gr3e6yiscria2
AWS instance: trn1.2xl

`
  File "<string>", line 111, in __init__
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 1341, in __post_init__
    and (get_xla_device_type(self.device) != "GPU")
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 127, in get_xla_device_type
    return xm.xla_real_devices([device])[0].split(":")[0]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 268, in xla_real_devices
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 268, in <listcomp>
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 263, in _xla_real_device
    raise RuntimeError('Invalid device format: {}'.format(device_str))
RuntimeError: Invalid device format: cpu
`

Python training script attached as txt format
[train-vit.txt](https://github.com/huggingface/transformers/files/12034074/train-vit.txt)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Create an EC2 Trn1 instance with [Hugging Face DLAMI ](https://aws.amazon.com/marketplace/pp/prodview-gr3e6yiscria2)
2. `pip install peft`
3. install optimum-neuron form source: `pip install git+https://github.com/huggingface/optimum-neuron.git`
4. run python3 train-vit.py (script attached)

### Expected behavior

To run training script to completion.
cc @pacman100 <|||||>Issue reported to https://github.com/huggingface/optimum-neuron/issues/134