Attempting to load non-existent vocab files from cache leads to a breaking behavior offline (while non-breaking online)
Hello, 

I found an error trying to use the  `openai/clip-vit-base-patch32` model offline, that was downloaded and cached from previous uses. Digging through the code, the problem seems to arise from the fact that when used online, attempting to download a file that does not exist  (here `added_tokens.json`) leads to a 404 Error, which simply sets the resolved_vocab_files[file_id] to None (line 1692 of `tokenization_utils_base.py`) with no further repercussions on the model use. 

However, using it offline, since the non-existent file was never found and thus never downloaded and cached, no file is matched by the `get_from_cache()` in `file_utils.py` and a ValueError is raised, preventing the use of the model offline ("Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.".)

Circumventing this ValueError similarly as what is done for the 404 leads to a perfectly working model (but I am guessing breaks in other cases). 

I am guessing possible solutions are to add a handcrafted exception for optional vocab files, or to cache a list of files that caused 404 during the initial download ?

Cheers, 
Manuel

## Environment info

- `transformers` version: 4.8.2
- Platform: Linux-5.8.0-59-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.3.4 (cpu)
- Jax version: 0.2.16
- JaxLib version: 0.1.68
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik is the most relevant for the issue I would say, as it is a general problem during offline tokenizer download.

## Information

Model I am using (Bert, XLNet ...):

I am using the `openai/clip-vit-base-patch32` but the bug would be reproducible with other models. Here the issue is caused by the non-existance of the `added_tokens.json`.

## To reproduce

Steps to reproduce the behavior:

1. Download the CLIP model with the default scripts.
2. Put computer offline.
3. Attempt to run the same scripts offline, loading the files from cache.

## Expected behavior

The expected behavior is for offline use of the model to work once cached (better described above). 

Hello! I am unsure of which scripts you are using. However, when instantiating a model or a tokenizer, you should specify 
the `local_files_only=True` parameter to the `from_pretrained` method when offline to ensure that the objects don't try to fetch files they can't access.<|||||>For every other file, they can be matched from cache even offline when  `local_files_only=False`. Since `added_tokens.json` was never downloaded cause non-existent (and thus raises a 404 which is a considered case), setting `local_files_only` to True  bypasses the `ValueError: Connection error ...` by instead raising a `FileNotFoundError` which is caught by the exception conditions l.1684 of `tokenization_utils_base.py`, and adds the non-existent file to the `unresolved_files` list, with a logger info message. On the other hand, the `ValueError` is not considered and interrupts program execution with an error message.

To allow users not to modify the `local_files_only` flag every time they go offline, wouldn't it be useful in this case to treat the `ValueError` the same way ? Adding it to the unresolved file list, logging a warning message and not letting the program interrupt itself for a failed resolution of a file that does not exist anyways ?

The scripts:
```
from PIL import Image
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32", local_files_only=False) 
# Setting flag to True breaks execution when offline - (ofc when tokenizer was already cached)

image = Image.open("/home/manu/perso/clip/harden.png").convert("RGB")
options = ["a photo of a man", "a photo of a cat"]
inputs = processor(text=options, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)
logits_per_image = outputs.logits_per_image
probs = logits_per_image.softmax(dim=1)
print(probs)
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.