Selecting specific GPU CUDA devices
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0
- Platform: Linux-4.15.0-65-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: 

### Who can help
Hello @sgugger,

Steps to reproduce the behavior:

1. I would like to use selected CUDA GPU cores among 8 of them in the HF `Trainer` class. I've written something along the following lines: 
2. So, I've done `export CUDA_VISIBLE_DEVICES=1,8` to select specific GPU devices, and ran:

```
training_args = TrainingArguments(
            output_dir=self._output_dir,
            overwrite_output_dir=True,
            num_train_epochs=epochs,
            per_device_train_batch_size=batch_size,
            per_device_eval_batch_size= int(batch_size/2), # since evaluation per 
            logging_steps = 20,
            save_total_limit = 20,
            warmup_steps=500, 
            weight_decay=0.01,
            evaluation_strategy = "steps",
            load_best_model_at_end = True,
            eval_accumulation_steps = 1,
            logging_dir = "logs"
        )
        
trainer = Trainer(
            model=self._model,
            args=training_args,
            tokenizer=self._tokenizer,
            data_collator=self._data_collator,
            train_dataset=self._train,
            eval_dataset = self._test,
        )
        
print("Devices used are:")
print(training_args.device)
```

## Expected behavior
I was under the impression that the `training_args.device` should return me cuda:1,8 or something along those lines, but it still reverted back to cuda:0. Are there any arguments I could specify to select a particular core.

Thank you in advance!

When you do `CUDA_VISIBLE_DEVICES=1,8`, CUDA will still call the two available GPUs 0 and 1, 0 will correspond to 1 and 1 to 8. I f you look at the output of `nvidia-smi`, you will see the training will only run on GPUs 1 and 8.<|||||>Thank you, silly me. I'll close the issue, thanks!