Cache the files in get_fast_tokenizer_file()
# 🚀 Feature request

`transformers.BertTokenizer.from_pretrained()` calls `get_fast_tokenizer_file()` which downloads a file from the HuggingFace server but never adds it to the cache.

Would be useful to cache that file (in the same folder as the models) to make CI runs more robust.

## Motivation

Try to avoid having issues such as the one below in our CI runs:
```
E           requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/models/mrm8488/bert-small-finetuned-squadv2
```

Hello! This does cache the file, but you're unfortunately hitting an error on our side when checking that the file exists and its sha (cc @julien-c, @n1t0). If this happens, I recommend leveraging `local_files_only` as a parameter to `from_pretrained`, as this will only use local files.

Sorry for the inconvenience.<|||||>AFAICT it always goes in here: https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1654
Then
https://github.com/huggingface/transformers/blob/c94c1b89674f2b15b23c8c4ce30f036bf883717f/src/transformers/tokenization_utils_base.py#L3486

It checks this https://github.com/huggingface/transformers/blob/c94c1b89674f2b15b23c8c4ce30f036bf883717f/src/transformers/file_utils.py#L2086
But that's in the current work directory, not the cache. (I think that might be the problem?)
And if it can't find it, it will download the files but not save them, so the following time it will re-download them again.<|||||>Yes, but this is only to get the name of the file to download. Once it has it (no download has occurred), it continues on to this code snippet:

https://github.com/huggingface/transformers/blob/c94c1b89674f2b15b23c8c4ce30f036bf883717f/src/transformers/tokenization_utils_base.py#L1688-L1704

It's using `cached_path`, so it should cache the file correctly!<|||||>I agree the vocab files are cached, however `get_list_of_files()` before that isn't and will always try to connect to the HuggingFace server.

We get several failures a day in our CI because of that specific call (Even though the other files are cached on the builder)<|||||>I think the error comes from the file check rather than the file download - the failure can definitely happen when checking the files on a remote repository, which I understand can make your CI fail.

We're in the process of implementing a retry mechanism on such issues that should partially solve this - thanks for raising the issue, and I hope we can deliver a more robust mechanism very soon.<|||||>Infra team is on it too (i.e on the occasional server side errors)<|||||>i would like have a mode that tries to do local_files_only but if files are missing it can still fetch them<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>This is still happening and still needs to be fixed