`No such file or directory` when setting `cache_dir`
### System Info

- `transformers` version: 4.26.1
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.9
- Huggingface_hub version: 0.13.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Note that `cache_model` directory exists**:

```
from transformers import CLIPProcessor, CLIPModel

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32", cache_dir="cache_model")


inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=None, return_tensors="np", padding=True)

print(inputs)
```

### Expected behavior

in recent versions of `transformers`, the code snippet above produces the following error:

```
FileNotFoundError: [Errno 2] No such file or directory: 'cache_model/models--openai--clip-vit-base-patch32/snapshots/e6a30b603a447e251fdaca1c3056b2a16cdfebeb/preprocessor_config.json'
```

whereas with `transformers==4.20.1`, the snippet successfully runs
I just tried your code sample and it runs without issue on both main and 4.26.1. Are you sure `"cache_model"` is in a folder you have write access to?<|||||>hi @sgugger, you are correct, I just made a fresh virtualenv and my snippet indeed does work. However, it doesn't work in my dev environment, nor in my projects CI, so I suspect theres an issue with a dependency somewhere?

I dug into what is happening inside `cache_model` in both environments, and found a clue:

In my new virtualenv:
```
lrwxr-xr-x  1 richard  staff   136B Mar 14 14:38 preprocessor_config.json -> /Users/richard/Projects/huggingface_bug/cache_model/models--openai--clip-vit-base-patch32/blobs/5a12a1eb250987a4eee0e3e7d7338c4b22724be1
```

in my dev environment:
```
lrwxr-xr-x  1 richard  staff    96B Mar 14 15:05 preprocessor_config.json -> cache_model/models--openai--clip-vit-base-patch32/blobs/5a12a1eb250987a4eee0e3e7d7338c4b22724be1
```

you can see that the former creates a symlink to an absolute path, whereas the latter uses a relative path, which may explain why it cannot find the actual file.

Do you have any ideas what could be causing this behaviour?<|||||>Might be a different version of `huggingface_hub`?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.