Trainer is using DataParallel on parallelized models 
## Environment info

- `transformers` version: 4.2.0
- Platform: Ubuntu 20.04
- Python version:  3.8.5
- PyTorch version (GPU?): 1.7.1 / CUDA 11.2
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@sgugger @stas00 


## Information

I'm trying out the 4.2.0 release with a training script that had been working in 4.1.1. 

I'm parallelizing my model over two GPUs, and I had been using the `--model_parallel` training arg in the previous version. Now that it's no longer used, I removed the arg from my training command, but I'm getting an error as though the DataParallel is being used and the model isn't being detected as parallelized:
`RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1`

I did some debugging, and everything seems okay with my model (`trainer. is_model_parallel` returns True). But the `trainer. args.n_gpu` is still 2.

I admit that I don't totally understand what's happening in the trainer code, but it might be an error on line 289?
[`self.args._n_gpu = 1`](https://github.com/huggingface/transformers/blob/126fd281bc309ec29caef99e982640265c8a4fba/src/transformers/trainer.py#L289)

Should that be `self.args.n_gpu = 1`, without the leading underscore?


## To reproduce

Steps to reproduce the behavior:

1. Parallelize a model
2. Train on a machine with multiple GPUs




The `self.args._n_gpu = 1` is to avoid parallelizing the data so it has nothing to do with your problem (and it is right, we can't set `self.args.n_gpu` which is a property but that's a whole different story!)

How is your model parallelized? Without that piece of code we can't reproduce the bug and help you.<|||||>Thanks @sgugger.

In my test, I'm using some code originally derived from the run_clm.py example. I'm trying to fine-tune a GPT2 model I've trained from scratch. The model was parallelized with the following lines, and this exact fine-tuning script ran successfully yesterday in 4.1.1, using the `--model_parallel` training arg. 

```
    device_map = {0: range(0, 15),
                  1: range(15, 32)}
    model.parallelize(device_map)
```

The error I'm getting now looks a lot like what would happen if I left out the `--model_parallel` flag in 4.1.1.
<|||||>> RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1

Please post the full trace.

I have only experimented with t5 and bart MP so far, but gpt2 is supposed to be very similar.

Most likely the outputs aren't being copied back to the 0th gpu on return, so this won't have anything to do with the trainer. Most likely the issue you encountered has to do with evaluation and not training.

I had to fix t5-MP to do that, but the PR with the fix hasn't been merged. 

https://github.com/huggingface/transformers/blob/58d047a596a97fbb815acb3e657102bf1960b06a/src/transformers/models/t5/modeling_t5.py#L1263-L1266

I won't be surprised if gpt2 is missing that too.

`model_parallel_inputs_to_specific_device` is a new function that isn't in master, but part of these 2 PRs: https://github.com/huggingface/transformers/pull/9323 and https://github.com/huggingface/transformers/pull/9384 - it relies on another function - the full new file is here: https://github.com/huggingface/transformers/blob/fe21c43745fcf3f7958c17c2ac461bd784094205/src/transformers/utils/model_parallel_utils.py

The current MP implementations are very limited and at the moment I highly recommend you look at DeepSpeed instead, see:
https://github.com/huggingface/transformers/issues/8771#issuecomment-759176685 and
https://github.com/huggingface/transformers/issues/8771#issuecomment-759248400
You will need master for that as it was just merged 2 days ago.

We also removed `--model_parallel` in trainer master as it wasn't fully baked in first place.<|||||>@stas00 This is linked to how `TrainingArguments.n_gpu` was computed. Could reproduce and test the fix in #9578 removes the bug.<|||||>That's easy then. The error though very much reminded me of the issue I described in my comment above.<|||||>Thanks both!

@stas00 Definitely excited to check out DeepSpeed – that's the reason I started testing my code in 4.2.0