[Bug]Non-robust directory splitting and detection at get_cached_module_file
### System Info

- `transformers` version: 4.30.2
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. I specified the correct model directory path under windows, but the library doesn't seem to find it correctly.
Just like this:
```python
# Directory THUDM/chatglm2-6b is existed at same directory with code.
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True)
```
2. Then I found the reason was that the `submodule` was not splitted correctly.
**Source Code At**: `transformers/dynamic_module_utils:get_cached_module_file`
```python
# line: 235 This code returns true on both linux and windows.
is_local = os.path.isdir(pretrained_model_name_or_path)
if is_local:
    # line: 237 But this code cann't split path correctly, because os.path.sep is "\\" rather than "/".
    submodule = pretrained_model_name_or_path.split(os.path.sep)[-1]
```
3. So I solved the problem by modifying my path.
```python
tokenizer = AutoTokenizer.from_pretrained("THUDM\\chatglm2-6b", trust_remote_code=True)
```

### Expected behavior

Even on windows code, it is common to specify paths by forward slashes. Although it may not be a standard paradigm.

So I think there are two possible approaches:
1. Use a more robust way to split `submodule`, such as the pathlib.
2. Explicitly throws a warning asking windows users to pass in a standard windows path.


Are you really certain `os.path.isdir(pretrained_model_name_or_path)` returns `True` with a path containing `/` on Windows? This seems really weird to me.<|||||>> Are you really certain `os.path.isdir(pretrained_model_name_or_path)` returns `True` with a path containing `/` on Windows? This seems really weird to me.

yes~
This is the screenshot of the debugging I just did on vscode.
![image](https://github.com/huggingface/transformers/assets/27395365/b623865f-a0cc-45db-9ed0-4f3f3a9c8517)
<|||||>Annoying (that it's super inconsistent like this). Will look at this later today and try to come up with a fix!<|||||>Could you try if the PR mentioned above fixes your issue?<|||||>> Could you try if the PR mentioned above fixes your issue?

No problem~<|||||>> Could you try if the PR mentioned above fixes your issue?

Problem has been solved by this PR! Should I close this issue now?<|||||>It will be closed auto-magically by GitHub when the PR is merged :-)