Connection error even though files are already downloaded.
### System Info

transformers version: 4.16.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run demo.py (from https://github.com/bes-dev/stable_diffusion.openvino) without having an internet connection, but the models are already downloaded.

### Expected behavior

When TRANSFORMERS_OFFLINE is not used, I get the following error:
```
Stacktrace:
  File "stable_diffusion_engine.py", line 26, in __init__
    self.tokenizer = CLIPTokenizer.from_pretrained(tokenizer)
  File "transformers/tokenization_utils_basepy", line 1707, in from_pretrained
    resolved_vocab_files[file_id] = cached_path(
  File "transformers/file_utils.py", line 1846, in cached_path
    output_path = get_from_cache(
  File "transformers/file_utils.py", line 2102, in get_from_cache
    raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.
```

When TRANSFORMERS_OFFLINE is set to 'true', it works as expected.
I expect it to work even if I don't use this environment variable.

Not sure if related, but I have also set the TRANSFORMERS_CACHE environment variable.
This should be fixed by https://github.com/huggingface/transformers/pull/19206, which we just released in a patch. Could you upgrade your transformers version and let me know if the issue is fixed? Thank you!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.