GPT-2 Model wrapped in DataParallel hangs immediately
## Environment info
- `transformers` version: 4.12.5
- Platform: Linux-3.10.0-1160.24.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.5
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True

### Who can help
@sgugger 

Models:
- GPT-2, GPT: @patrickvonplaten, @LysandreJik

## Information

Process hangs indefinitely and cannot be killed. Afterwards, other processes that try to interact with the GPUs (e.g. `nvidia-smi`) become stuck/uninterruptible as well. Node then requires a restart. Problem does not occur when the DataParallel wrapper is removed from script below.

GPUs: 4 x Nvidia Quatro RTX 5000
CUDA 11.3 (also tested with 10.2)

Model I am using: GPT2-medium

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

Py-Spy Output:
```Python
Thread 10623 (active): "MainThread"
    adam (torch/optim/_functional.py:87)
    step (torch/optim/adam.py:133)
    decorate_context (torch/autograd/grad_mode.py:28)
    wrapper (torch/optim/optimizer.py:88)
    main (minimal_gpu.py:38)
    <module> (minimal_gpu.py:43)
Thread 10689 (idle)
Thread 10688 (idle)
Thread 10690 (idle)
Thread 10691 (idle)

```

## To reproduce

Steps to reproduce the behavior:

run
```python
import torch 
import transformers

device = 'cuda' if torch.cuda.is_available() else 'cpu'


def main():
    ''' python minimal_gpu.py'''

    # load tokenizer an model
    tokenizer = transformers.GPT2Tokenizer.from_pretrained("gpt2-medium")
    model = transformers.GPT2LMHeadModel.from_pretrained("gpt2-medium")

    model = torch.nn.DataParallel(model)
    model = model.to(device)

    # load dataset
    dataset = tokenizer(["This is a test sentence",
                         "And so is this one",
                         "One two three four banana"], return_tensors="pt").to(device)
    dataset.update({"labels": dataset["input_ids"]})

    # set up optimizer
    optimizer = torch.optim.Adam(params=model.parameters(), lr=0.00001)

    # forward pass
    outputs = model(**dataset)  # <----- gets stuck here
    loss = outputs[0]

    # backward pass and update weights
    optimizer.zero_grad()
    loss.sum().backward()
    optimizer.step()



if __name__ == '__main__':
    main()


```
## Expected behavior

Model gets trained on dummy data.

The script executes without any problem on my side (two Titans, PyTorch 1.10 and CUDA 10.2), so there is nothing wrong with the model per se. How are you launching the script?<|||||>I was afraid that would be the case and I'm facing a driver issue or something. I am launching simply with `python script.py`, where script.py contains the code above. Same thing happens when I run this from an interactive shell.

How would I debug this? This has cost me quite some time already, this is really becoming an issue.<|||||>Did you try `DistributedDataParallel`? It might work and avoid you the debugging.
Otherwise checking you can do a simple gather of tensors might be a nice first step (but that will require launching the script in distributed mode AFAIK).<|||||>So, running it on a different server worked fine. There must be something wrong at a different level, maybe at some point I can come back to it and figure it out.

Anyway, with DataParallel I can't use AMP anymore, which reduces the sample size we can run by about 20 percent. I saw that it should work if the forward method of the model is wrapped in autocast, but how can I do that with a pretrained GPT-2 model?

I tried running DDP as well. It seems to work on the server where DP doesn't work, but not without problems. The biggest one is that utilization is so low that it is slower than using a single GPU (I don't even know how to do profiling with multiprocessing and multiple GPUs). The other one is that it crashes during evaluation, but only on the last evaluation pass (evaluation is done every half epoch). Except if I reduce the size of the train dataset, then it crashes during the first evaluation pass already. It's really odd. This happens with gloo as the backend - with nccl it hangs forever at initialization. I'll open a new ticket about this at some point, if I can come up with a minimal version that reproduces the problem.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.