BertTokenizer.from_pretrained fails for local_files_only=True when added_tokens.json is missing
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.1
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.6.1810-Core
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@mfuntowicz
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): `google/bert_uncased_L-2_H-128_A-2`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Run the following:
```
from transformers import BertTokenizer
BertTokenizer.from_pretrained('google/bert_uncased_L-2_H-128_A-2')
BertTokenizer.from_pretrained('google/bert_uncased_L-2_H-128_A-2', local_files_only=True)
```

In the Python interpreter, this produces the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gscratch/cse/julianjm/anaconda3/lib/python3.7/site-packages/transformers-4.0.1-py3.8.egg/transformers/tokenization_utils_base.py", line 1747, in from_pretrained
  File "/gscratch/cse/julianjm/anaconda3/lib/python3.7/site-packages/transformers-4.0.1-py3.8.egg/transformers/file_utils.py", line 1007, in cached_path
  File "/gscratch/cse/julianjm/anaconda3/lib/python3.7/site-packages/transformers-4.0.1-py3.8.egg/transformers/file_utils.py", line 1171, in get_from_cache
ValueError: Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.
```

Looking more closely, I have isolated the issue to the logic [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1774). In this case, the error is because the cached path for the url `https://huggingface.co/google/bert_uncased_L-2_H-128_A-2/resolve/main/added_tokens.json` cannot be found in the cache when `local_files_only=True`. This is because the URL 404s; i.e., the file does not exist.

When `local_files_only=False`, the GET returns a 404 and the tokenizer init code just ignores the missing file. However, when `local_files_only=True` and the file is not found, it throws a `ValueError` instead which is not caught.

What makes this non-trivial is that without making HTTP requests, there is no way of telling the difference between a file that doesn't exist and a file which exists but hasn't been downloaded. It seems to me that there are several potential ways of fixing the issue.

1. Ensure that all files exist. Don't let people upload incomplete sets of files (and fix the ones which are currently incomplete).
2. Recover from 404s by caching an "empty" file here. But this only works where there is a meaningful notion of "empty" file, like lists of tokens. I think this would not work for json files or serialized models.
3. Put a special kind of file in the cache which says "hey, this file isn't supposed to exist", and handle appropriately everywhere files are loaded. Potentially could throw a special error saying the file isn't supposed to exist; HTTP 404s could then be caught and re-thrown as this special error, so, the case could be handled uniformly.
4. Just log a warning for files that aren't in the cache, and treat them like 404s. Wild west, but at least if the code unexpectedly fails later the user will be able to guess the problem. Easy to implement, but will worsen the UX every time someone tries to use `local_files_only` without downloading the model first.

Option 3 seems the cleanest to me, while option 4 is what I'm shunting into my transformers egg for now so I can keep working.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

After downloading, I would expect any artifact to be loadable from cache and equivalent to the downloaded one.

<!-- A clear and concise description of what you would expect to happen. -->

Actually, all of the files 404 here except `vocab.txt`. I have `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`, and `tokenizer.json` all missing for this model.<|||||>> Actually, all of the files 404 here except `vocab.txt`. I have `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`, and `tokenizer.json` all missing for this model.

If these files are missing even BertTokenizer.from_pretrained('google/bert_uncased_L-2_H-128_A-2'); should give an error; however it passed due to the below code; any particular reason this logic was added in the below mentioned:

https://github.com/huggingface/transformers/blob/master/src/transformers/file_utils.py#L1232<|||||>@hlahkar Are you sure? The code you linked seems to just check for `requests.exceptions.ConnectionError` and `requests.exceptions.Timeout`. I think a 404 will raise a `requests.exceptions.HTTPError`, which bubble up to be thrown by `get_from_cache`, through `cached_path`, and then [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1774) where it is then caught and ignored.

In fact, my hacky workaround was to replace [this line](https://github.com/huggingface/transformers/blob/master/src/transformers/file_utils.py#L1257) with `raise requests.exceptions.HTTPError("404 Client Error")`, so the same thing happens when `local_files_only=True`; now I can load the tokenizer in that case.<|||||>> @hlahkar Are you sure? The code you linked seems to just check for `requests.exceptions.ConnectionError` and `requests.exceptions.Timeout`. I think a 404 will raise a `requests.exceptions.HTTPError`, which bubble up to be thrown by `get_from_cache`, through `cached_path`, and then [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1774) where it is then caught and ignored.
> 
> In fact, my hacky workaround was to replace [this line](https://github.com/huggingface/transformers/blob/master/src/transformers/file_utils.py#L1257) with `raise requests.exceptions.HTTPError("404 Client Error")`, so the same thing happens when `local_files_only=True`; now I can load the tokenizer in that case.

My concern is should we also not be going into the error flow whenever we are getting a 404 error also; otherwise it might give a false sense of working to the user<|||||>In my previous comment, I mentioned the wrong line number. My Question is; why is the 404 error ignored in the below code segment:
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1784<|||||>So, is this problem solved in any way? 
It seems it is now impossible to use most Bert-like models without the Internet connection, even though all the model files are cached.
Transformers tries to get the `added_tokens.json` file, can't find it, and fails with "ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on."
This is really bothersome on HPC systems, where compute nodes are often offline by design.<|||||>@akutuzov on which version of transformers are you?

I agree that this is a bug that we should solve, cc @LysandreJik @sgugger <|||||>Taking a look.<|||||>@julien-c I use Transformers 4.1.1<|||||>Aimed to fix that in #9807, feedback appreciated @julianmichael <|||||>The PR looks good as a stopgap — I guess the subsequent check [at L1766](https://github.com/huggingface/transformers/pull/9807/files#diff-85b29486a884f445b1014a26fecfb189141f2e6b09f4ae701ee758a754fddcc1R1766) will catch the case where the tokenizer hasn't been downloaded yet since no files should be present. But is this problem necessarily only for tokenizers? It seems like a general issue which is going to hold for any cached resources that have optional files. It might be cleaner to handle it in the file cache itself. But that's a much bigger issue I guess.<|||||>I believe this is only the case for tokenizers. The two other that could be possibly affected by this are:
- Configuration downloads -> downloads a single file
- Model downloads -> downloads the configuration file and the model state dict, both of which are necessary and need to raise an error if missing.

Let me know if you think I'm missing something and I'll see what we can do. <|||||>Ok, sounds good. No need for unnecessary/premature refactoring then :)