Loading from the wrong cache? 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.12.5
- Platform:linux
- Python version:3.8
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help

@LysandreJik

## Information

Model I am using (Bert, XLNet ...):'google/t5-v1_1-small'

The problem arises when loading a tokenizer

## To reproduce

Steps to reproduce the behavior:

1. Two cache dirs one you don't have access to (coworker's)
2. load
3.

```
$ tokenizer_name
'google/t5-v1_1-small'
$ cache_dir
'/outputs/.cachetmp/'
$ AutoTokenizer.from_pretrained(tokenizer_name, use_fast=True, cache_dir=cache_dir)
loading configuration file https:/huggingface.co/google/t5-v1_1-small/resolve/main/config.json from cache at /outputs/.cachetmp/64521636c162517d8b5b18cbb5b1eda52138a4e70ab9de6f1f996bc3c668233f.e890bc92245b637cd45d26aae3f8c93f29e9d928c73b2e7e1007db14eb98c948
Model config T5Config {
  "architectures": [
    "T5ForConditionalGeneration"
  ],
  "d_ff": 1024,
  "d_kv": 64,
  "d_model": 512,
  "decoder_start_token_id": 0,
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "gated-gelu",
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "layer_norm_epsilon": 1e-06,
  "model_type": "t5",
  "num_decoder_layers": 8,
  "num_heads": 6,
  "num_layers": 8,
  "output_past": true,
  "pad_token_id": 0,
  "relative_attention_num_buckets": 32,
  "tie_word_embeddings": false,
  "transformers_version": "4.12.5",
  "use_cache": true,
  "vocab_size": 32128
}
loading file https:/huggingface.co/google/t5-v1_1-small/resolve/main/spiece.model from cache at /outputs/.cachetmp/c8b0274da206819bf0ed2bd3b928edd6b170794dfd2c9186b7f4d8182c89a289.3b69006860e7b5d0a63ffdddc01ddcd6b7c318a6f4fd793596552c741734c62d
loading file https:/huggingface.co/google/t5-v1_1-small/resolve/main/tokenizer.json from cache at None
loading file https:/huggingface.co/google/t5-v1_1-small/resolve/main/added_tokens.json from cache at None
loading file https:/huggingface.co/google/t5-v1_1-small/resolve/main/special_tokens_map.json from cache at /outputs/.cachetmp/3ad6f8335c1b1ef8966245899d47dcf735abd134d21fd7d26f621fe45ac01184.c94798918c92ded6aeef2d2f0e666d2cc4145eca1aa6e1336fde07f2e13e2f46
loading file https:/huggingface.co/google/t5-v1_1-small/resolve/main/tokenizer_config.json from cache at /outputs/.cachetmp/385731228f2b42821bd28ef2bcab8b6c77982e1da41d17058802b306f5068ada.b1a2e3c152960fdc6b3d16520fa9f1591e2818d7dd66946c219e651f224894bf
[Errno 13] Permission denied: '**/.cache/huggingface/**64521636c162517d8b5b18cbb5b1eda52138a4e70ab9de6f1f996bc3c668233f.e890bc92245b637cd45d26aae3f8c93f29e9d928c73b2e7e1007db14eb98c948.lock'
Traceback (most recent call last):
  File "/u/leshemc/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/anaconda3/envs/fuse/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 505, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/anaconda3/envs/fuse/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1744, in from_pretrained
    return cls._from_pretrained(
  File "/anaconda3/envs/fuse/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1872, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/anaconda3/envs/fuse/lib/python3.8/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 128, in __init__
    super().__init__(
  File "/anaconda3/envs/fuse/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__
    raise ValueError(
ValueError: Couldn't instantiate the backend tokenizer from one of: 
(1) a `tokenizers` library serialization file, 
(2) a slow tokenizer instance to convert or 
(3) an equivalent slow tokenizer class to instantiate and convert. 
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.```

## Expected behavior
Load the tokenizer and ignore the other cache dir, how did it even find out this dir exists

If it helps, I see that the first cache_dir works well, but for some reason when "url_or_filename== 'https://huggingface.co/google/t5-v1_1-small/resolve/main/config.json'" the cache dir passed is None, and then it looks in the wrong cache and breaks. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.