TrainingArguments initializer changes `torch.current_device()`
### System Info

```
transformers 4.28.1
python 3.9.13
torch 1.12.1
```

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run
```python
import torch
from transformers import TrainingArguments
torch.cuda.set_device(1)
print(torch.cuda.current_device())
training_args = TrainingArguments(output_dir="output/")
print(torch.cuda.current_device(), training_args.device)
```
Observe
```
1
0 cuda:0
```

### Expected behavior

I would expect that the `torch.current_device()` would not change and therefore to observe
```
1
1 cuda:1
```
Your expectation is not correct as `TrainingArguments` will take all GPUs available (in this case it will use DataParallel on your model for the training afterward). You need to set the `CUDA_VISIBLE_DEVICES` environment variable to limit the GPUs seen.<|||||>Oh, I see. Thanks for the answer!