Computer shuts down when multiple gpus are used
## Environment info
- transformers version: 4.10.3
- Platform: Linux-4.15.0-159-generic-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyTorch version (GPU?): 1.9.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes (through default settings)
- 
### Who can help
Library:
- trainer: @sgugger
- 
## Information
Model I am using: AutoModelForSequenceClassification
The problem arises when using:
* [x] the official example scripts:
https://huggingface.co/transformers/training.html#fine-tuning-in-pytorch-with-the-trainer-api

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: IMDB task


## To reproduce
Steps to reproduce the behavior:
here is an implementation that was run

```
from transformers import TrainingArguments
from transformers import Trainer
from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer
from datasets import load_from_disk

device = torch.device('cuda:0')
raw_datasets = load_from_disk('imdb')
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
model = AutoModelForSequenceClassification.from_pretrained('bert-base-cased', num_labels=2)

def tokenize_function(examples):
  tokenized = tokenizer(
    examples["text"], padding="max_length", truncation=True)
  return tokenized

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
tokenized_datasets.set_format('torch', columns=[
               'attention_mask', 'input_ids', 'label', 'token_type_ids'])

small_train_dataset = tokenized_datasets["train"].shuffle(
  seed=42).select(range(10))
small_eval_dataset = tokenized_datasets["test"].shuffle(
  seed=42).select(range(10))

training_args = TrainingArguments("test_trainer")

model.to(device=device)

trainer = Trainer(
  model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()
```

As the trainer starts to train, the computer shuts off. The issue was observed only when there were 2 gpus (both Titan rtx). It doesn't seem to be a load or power issue, because the issue occurs even with very small datasizes.

Through some testing, I suspect that the issue is occurring when the trainer is trying to use both gpus. Since the trainer's default behavior is to use both gpus if possible, I was able to avoid the issue by using CUDA_VISIBLE_DEVICES =0, so that the trainer does not attempt to send data to both gpus. This prevents the issue, but also prevents the system from using both GPUs.

The computer shut off before producing any meaningful log message. It's very odd that the system is struggling with such a small load from a simple test code. Any help in diagnosing this problem would be very much appreciated.

It is incredibly unlikely that this is caused directly by `transformers` code, or any specific code for that matter. It is much more likely that this is a hardware issue. First thing that comes to mind is that your PSU cannot handle the spike when initializing, or that you have not used separate V lanes for each GPU.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.