Can't lazy initialize BART model on GPU
## Environment info
- `transformers` version: 3.5.1
- Platform: Linux-4.14.81.bm.15-amd64-x86_64-with-debian-9.11
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
 Bart: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Bart

We have finetuned a BART model for Seq2SeqLM, and want to serve it using a Python web microframework. This microframework uses `multiprocessing` underneath, so we wish to lazily initialize the model on the GPU on the first call. However, in `modeling_bart.py`, some of the layers already call CUDA in the main process:

https://github.com/huggingface/transformers/blob/f38c4ad302dd34faef1137b13e9636f4408b0462/src/transformers/models/bart/modeling_bart.py#L125

So trying to do so results in the following error:

```
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```

## To reproduce

```
class Model:
    def __init__(self, path):
        self.path = path
        self.initialized = False

    def init(self):
        if not self.initialized:
            self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
            self.model = AutoModelForSeq2SeqLM.from_pretrained(self.path)
            self.model.to(self.device)
            self.model.eval()
            self.initialized = True
```

```
model = Model("bart-base")
# in multiprocessing subprocess...
model.init()

# RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```

Sorry about the pseudocode, the microframework used is proprietary.

Commenting out the linked lines removes the error.

## Expected behavior

Able to call `model.cuda()` from subprocess.
@jethrokuan - Please try this on the transformers 4.1.0; issue looks to be fixed in this version<|||||>Actually it looks like the problem is exasperated in newer HF. In a devbox with no CUDA:

```
Python 3.7.9 (default, Nov 18 2020, 11:18:33)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
.../lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
```
  
This suggests that the simple action of importing anything from hf causes cuda initialization.

I worked around it by deferring the import altogether:

```
class Model:
    def __init__(self, path):
        self.path = path
        self.initialized = False

    def init(self):
        if not self.initialized:
            from transformers import ...
            self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
            self.model = AutoModelForSeq2SeqLM.from_pretrained(self.path)
            self.model.to(self.device)
            self.model.eval()
            self.initialized = True
```
  