Zero Shot Classification Pipeline fails when running in CPU-only Docker container
## Environment info
- `transformers` version: 4.0.0
- Platform: MacOS 10.15.7 (2018 MacBook Pro 15")
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0 CPU Only
- Tensorflow version (GPU?): N/A
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help
Maybe @LysandreJik ?

## Information

Model I am using (Bert, XLNet ...): facebook/bart-large-mnli

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Pull and start the official [transformers-pytorch-cpu](https://hub.docker.com/r/huggingface/transformers-pytorch-cpu/dockerfile) container.
2. `docker exec -it huggingface bash`
3. `python3`
4. `from transformers import pipeline`
5. `classifier = pipeline("zero-shot-classification", model='facebook/bart-large-mnli', tokenizer='facebook/bart-large-mnli', device=-1)`

Step 4 above results in the following warning being printed:
```
/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
```

Step 5 above results in the models being downloaded, then `Killed` is printed and the Python interpreter exits.

## Expected behavior
When running locally in a Jupyter Notebook or directly in the terminal (not in a container), the following works correctly and the warning about the CUDA initialization isn't printed:
```
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model='facebook/bart-large-mnli', tokenizer='facebook/bart-large-mnli', device=-1)
```

The problem seems to be limited to either the zero shot classification pipeline, or the facebook/bart-large-mnli model, since the following works correctly in the container (though the warning from Step 4 about the CUDA initialization is still printed):
```
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.