Multiprocessing behavior change 3.1.0 -> 3.2.0
## Environment info

```
- `transformers` version: 3.2.0
- Platform: Linux-4.15.0-88-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes (Python multiprocessing)
```

## Information
I am writing a custom script that uses Python's multiprocessing. The goal of it is to have multiple child processes that run inference (using `torch.nn.Module`) **on separate GPUs**.

See below a minimal example of the issue. Please note that script contains pure `torch` code, however, it seems like importing `transformers` (**and not even using it afterwards**) changes some internal states.

```python
import multiprocessing as mp
import os

import torch
import transformers  # <-- Just imported, never used

def diagnostics(name):
    """Print diagnostics."""
    print(name)
    print(f"CUDA initialized: {torch.cuda.is_initialized()}")
    print(f"Is bad fork: {torch._C._cuda_isInBadFork()}")
    print(80 * "*")

def fun(gpu):
    current_process = mp.current_process()

    diagnostics(current_process.pid)
    os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu)
    model = torch.nn.Linear(200, 300)

    model = model.to("cuda")  # Trouble maker

    while True:
        model(torch.ones(32, 200, device="cuda"))

if __name__ == "__main__":
    n_processes = 2
    gpus = [0, 1]
    start_method = "fork"  # fork, forkserver, spawn

    diagnostics("Parent")

    mp.set_start_method(start_method)

    processes = []
    for i in range(n_processes):
        p = mp.Process(name=str(i), target=fun, kwargs={"gpu": gpus[i]})
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

```
The above script works as expected in `3.1.0` or when we do not import transformers at all. Each subprocess does inference on a separate GPU. See below the standard output.
```
Parent
CUDA initialized: False
Is bad fork: False
********************************************************************************
21091
CUDA initialized: False
Is bad fork: False
********************************************************************************
21092
CUDA initialized: False
Is bad fork: False
********************************************************************************
```

 However, for `3.2.0` and higher there is the following error.
```
Parent
CUDA initialized: False
Is bad fork: False
********************************************************************************
21236
CUDA initialized: False
Is bad fork: True
********************************************************************************
Process 0:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "git_example.py", line 21, in fun
    model = model.to("cuda")  # Trouble maker
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 610, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 164, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
21237
CUDA initialized: False
Is bad fork: True
********************************************************************************
Process 1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "git_example.py", line 21, in fun
    model = model.to("cuda")  # Trouble maker
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 610, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 164, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```

Changing the `start_method` to `"forkserver"` or `"spawn"` prevents the exception from being raised. However, only a single GPU for all child processes is used.


Sorry to insist. Could you at least share your thoughts on this? @LysandreJik @patrickvonplaten <|||||>Is the only dependency you change `transformers`? The PyTorch version remained the same (v.1.7.0) for both `transformers` versions? If so, I'll take a deeper look this week.<|||||>@LysandreJik In both experiments `torch==v1.7.0`. However,  downgrading to `torch==v.1.6.0` (in both experiments) leads to exactly the same problem. I did `pip install transformers==v3.1.0` and `pip install transformers==v3.2.0` back and forth so that could be the only way other dependencies got updated.

Thank you!<|||||>Okay, thanks for checking. I'll have a look this week.<|||||>Hello @LysandreJik , is there any update on this issue?
Thank in advance!<|||||>This issue has been stale for 1 month.<|||||>I think I managed to solve the problem.

Instead of using the environment variable `os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu)` to specify the GPUs one needs to provide it via `torch.device(f"cuda:{gpu}")`.