Can't load previously built tokenizers
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.1
- Platform: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: not for this part that triggers this error
- Using distributed or parallel set-up in script?: n

### Who can help
Probably @mfuntowicz  or @patrickvonplaten 

## Information
N/A -- none of the fields here applied

## To reproduce
Context: I work on cluster where most nodes don't have internet access. Therefore I pre-build tokenizers, models, etc., in cli on nodes with internet access and then make sure that I can access the local caches on other nodes. That last part -- accessing the tokenizer I've built -- is failing for BlenderBot 400M distilled tokenizer. It's also failing for blenderbot small 90M which I also built today, potentially for others too, but it doesn't seem to be failing for roberta-base, which I had built before (and is a tokenizer small rather than base).
1. `AutoTokenizer.from_pretrained('facebook/blenderbot-400M-distill')` from a node with internet access
2. the same as above, from a node without internet access
3. You should see this error getting triggered : [https://github.com/huggingface/transformers/blob/14d677ca4a62facf70b28f2922b12e6cd3692a03/src/transformers/file_utils.py#L1234](url)


Here's the specific Traceback:
`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lusers/margsli/miniconda3/envs/latest/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 388, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/lusers/margsli/miniconda3/envs/latest/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1738, in from_pretrained
    resolved_vocab_files[file_id] = cached_path(
  File "/usr/lusers/margsli/miniconda3/envs/latest/lib/python3.8/site-packages/transformers/file_utils.py", line 1048, in cached_path
    output_path = get_from_cache(
  File "/usr/lusers/margsli/miniconda3/envs/latest/lib/python3.8/site-packages/transformers/file_utils.py", line 1234, in get_from_cache
    raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.`

Dug around a little and found that cached_path() gets called 7 filenames/urls when I'm offline and only 6 when I'm online (I printed cache_path every time cached_path() gets called) -- the last one is not seen when offline, and that's the one that triggers the error.  Printed the same things for other tokenizers I had previously built and didn't see this. Not sure if that's helpful, but it was as far as I got during my debugging.


## Expected behavior

no error

Hello! To make sure I understand your issue, you're doing the following:

```py
AutoTokenizer.from_pretrained('facebook/blenderbot-400M-distill')
```

on a node which has internet access, and then you're doing the same once you have no internet access. You want the library to rely on the cache that it had previously downloaded, is that right?

Could you make sure you are up to date with the `master` branch, and try the following once you have no internet access:

```py
AutoTokenizer.from_pretrained('facebook/blenderbot-400M-distill', local_files_only=True)
```

Thank you.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.