Offline mode not working for remote code?
### System Info

I want to run remote code offline and the revision is in my cache dir. For example,

```python
from transformers import AutoConfig
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, revision="cde457b39fe0670b10dd293909aab17387ea2c80", local_files_only=True)
```

However, it still reports that connection error.

```
ConnectionError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/THUDM/chatglm-6b/revision/cde457b39fe0670b10dd293909aab17387ea2c80 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6f8f9887f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
```

Is there anything wrong for offline mode with remote code?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run this piece of code offline:

```python
from transformers import AutoConfig
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, revision="cde457b39fe0670b10dd293909aab17387ea2c80", local_files_only=True)
```

### Expected behavior

run remote code offline with local cache
This has just been fixed on the main branch (by #22661) so make sure to use the latest!<|||||>It works for normal cases. However, when I set my customized cache dir, it still report error:

```
    445         f" cached files and it looks like {path_or_repo_id} is not the path to a directory containing 
a file named"                                                                                                 
    446         f" {full_filename}.\nCheckout your internet connection or see how to run the library in offlin
e mode at"                                                                                                    
    447         " 'https://huggingface.co/docs/transformers/installation#offline-mode'."                      
    448     )                                                                                                 
    449 except EntryNotFoundError:                                                                            
    450     if not _raise_exceptions_for_missing_entries:                                                     
                                                                                                              
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached fil
es and it looks like THUDM/chatglm-6b is not the path to a directory containing a file named config.json.     
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'. 
```

I just set `export TRANSFORMERS_CACHE=/my/cache/dir`.<|||||>Please give us a reproducible code example as well as the full traceback.<|||||>First, run the following code to download cache:

```python
from transformers import AutoConfig
config = AutoConfig.from_pretrained('THUDM/chatglm-6b', trust_remote_code=True, revision="aa51e62ddc9c9f334858b0af44cf59b05c70148a")
```

Then, run the same code with `TRANSFORMERS_OFFLINE=1 TRANSFORMERS_CACHE=~/.cache/huggingface` environment variables. Things will go wrong.<|||||>That's probably because you are not using the right folder. The default cache folder is in `~/.cache/huggingface/hub` so executing the lines above with `TRANSFORMERS_OFFLINE=1 TRANSFORMERS_CACHE=~/.cache/huggingface` doesn't work on my side but `TRANSFORMERS_OFFLINE=1 TRANSFORMERS_CACHE=~/.cache/huggingface/hub` does.<|||||>wow, that's cool. It works now. Thank you so much!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.