PRs error which occurs many times in the last days
## 🐛 Bug

I've seen that many recent PRs (e.g. [2201](https://github.com/huggingface/transformers/pull/2201), [2203](https://github.com/huggingface/transformers/pull/2203), [2190](https://github.com/huggingface/transformers/pull/2190), [2189](https://github.com/huggingface/transformers/pull/2189), ...) have encountered the same error reported below. All the PRs I've mentioned above passed 7 different checks, but not the **`ci/circleci: build_py3_torch_and_tf`**.
```
python -m pytest -sv ./transformers/tests/ --cov
============================= test session starts ==============================
platform linux -- Python 3.5.9, pytest-5.3.2, py-1.8.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /home/circleci/transformers
plugins: cov-2.8.1
collecting ... Fatal Python error: Segmentation fault

Current thread 0x00007f0afaba9740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 938 in create_module
  File "<frozen importlib._bootstrap>", line 577 in module_from_spec
  File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/usr/local/lib/python3.5/site-packages/torch/__init__.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 697 in exec_module
  File "<frozen importlib._bootstrap>", line 673 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/home/circleci/transformers/transformers/file_utils.py", line 38 in <module>
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 697 in exec_module
  File "<frozen importlib._bootstrap>", line 673 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/home/circleci/transformers/transformers/__init__.py", line 20 in <module>
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 697 in exec_module
  File "<frozen importlib._bootstrap>", line 673 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 943 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 943 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/usr/local/lib/python3.5/site-packages/py/_path/local.py", line 701 in pyimport
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 492 in _importtestmodule
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 424 in _getobj
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 248 in obj
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 440 in _inject_setup_module_fixture
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 427 in collect
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 254 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 234 in from_call
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 254 in pytest_make_collect_report
  File "/usr/local/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 373 in collect_one_node
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 717 in genitems
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 720 in genitems
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 492 in _perform_collect
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 454 in perform_collect
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 255 in pytest_collection
  File "/usr/local/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 245 in _main
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 196 in wrap_session
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
  File "/usr/local/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/site-packages/_pytest/config/__init__.py", line 92 in main
  File "/usr/local/lib/python3.5/site-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/local/lib/python3.5/runpy.py", line 85 in _run_code
  File "/usr/local/lib/python3.5/runpy.py", line 193 in _run_module_as_main

Received "segmentation fault" signal
```
## Expected behavior

When a new PR comes into Transformers, it does not generate this bug. 

## Environment

* OS: **Ubuntu 16.04**
* Python version: **3.6.9**
* PyTorch version: **1.3.1**
* PyTorch Transformers version (or branch): **master** (installed with `pip install transformers`)
* Using GPU **Indifferent**
* Distributed of parallel setup **Indifferent**
* Any other relevant information:
#2207 <|||||>Solved by @LysandreJik yesterday (this was due to upstream dependency bug)