`AutoTokenizer.from_pretrained` raise error when another same filename tokenizer imported
### System Info

- `transformers` version: 4.28.1
- Platform: Linux-4.19.91-009.ali4000.alios7.x86_64-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.2
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. bug reproduction
```python
from transformers import AutoTokenizer

# import one tokenizer from local (/home/admin/notebook/THUDM/chatglm2-6b/tokenization_chatglm.py)
tokenizer = AutoTokenizer.from_pretrained("/home/admin/notebook/THUDM/chatglm2-6b/", trust_remote_code=True)

# import another version tokenizer from another directory, but same filename (/home/admin/notebook/THUDM/chatglm-6b/tokenization_chatglm.py)
tokenizer = AutoTokenizer.from_pretrained("/home/admin/notebook/THUDM/chatglm-6b/", trust_remote_code=True)
```

raise exception 
```text
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 10
      6 from transformers import AutoTokenizer
      8 tokenizer = AutoTokenizer.from_pretrained("/home/admin/notebook/THUDM/chatglm2-6b/", trust_remote_code=True)
---> 10 tokenizer = AutoTokenizer.from_pretrained("/home/admin/notebook/THUDM/chatglm-6b/", trust_remote_code=True)

File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py:702, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    698     if tokenizer_class is None:
    699         raise ValueError(
    700             f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
    701         )
--> 702     return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    704 # Otherwise we have to be creative.
    705 # if model is an encoder decoder, the encoder tokenizer class is used by default
    706 if isinstance(config, EncoderDecoderConfig):

File /usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py:1811, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1808     else:
   1809         logger.info(f"loading file {file_path} from cache at {resolved_vocab_files[file_id]}")
-> 1811 return cls._from_pretrained(
   1812     resolved_vocab_files,
   1813     pretrained_model_name_or_path,
   1814     init_configuration,
   1815     *init_inputs,
   1816     use_auth_token=use_auth_token,
   1817     cache_dir=cache_dir,
   1818     local_files_only=local_files_only,
   1819     _commit_hash=commit_hash,
   1820     **kwargs,
   1821 )

File /usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py:1965, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, use_auth_token, cache_dir, local_files_only, _commit_hash, *init_inputs, **kwargs)
   1963 # Instantiate tokenizer.
   1964 try:
-> 1965     tokenizer = cls(*init_inputs, **init_kwargs)
   1966 except OSError:
   1967     raise OSError(
   1968         "Unable to load vocabulary from file. "
   1969         "Please check that the provided vocabulary is accessible and not corrupted."
   1970     )

File ~/.cache/huggingface/modules/transformers_modules/tokenization_chatglm.py:69, in __init__(self, vocab_file, padding_side, **kwargs)
     66 def _get_text_tokenizer(self):
     67     return self.text_tokenizer
---> 69 @staticmethod
     70 def get_blank_token(length: int):
     71     assert length >= 2
     72     return f"<|blank_{length}|>"

File /usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils.py:347, in PreTrainedTokenizer.__init__(self, **kwargs)
    346 def __init__(self, **kwargs):
--> 347     super().__init__(**kwargs)
    349     # Added tokens - We store this for both slow and fast tokenizers
    350     # until the serialization of Fast tokenizers is updated
    351     self.added_tokens_encoder: Dict[str, int] = {}

File /usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py:1534, in PreTrainedTokenizerBase.__init__(self, **kwargs)
   1530 self.deprecation_warnings = (
   1531     {}
   1532 )  # Use to store when we have already noticed a deprecation warning (avoid overlogging).
   1533 self._in_target_context_manager = False
-> 1534 super().__init__(**kwargs)

File /usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py:828, in SpecialTokensMixin.__init__(self, verbose, **kwargs)
    826     setattr(self, key, value)
    827 elif isinstance(value, (str, AddedToken)):
--> 828     setattr(self, key, value)
    829 else:
    830     raise TypeError(f"special token {key} has to be either str or AddedToken but got: {type(value)}")

AttributeError: can't set attribute
```

In this traceback, I found the exception raised from a abnormal line 69 `@staticmethod`, which pointed to the correct file `/home/admin/notebook/THUDM/chatglm-6b/tokenization_chatglm.py`
![image](https://github.com/huggingface/transformers/assets/25120867/1762dd64-8d92-49f4-963e-5ad26dece4d5)

but the exception was raised from the wrong file (/home/admin/notebook/THUDM/chatglm2-6b/tokenization_chatglm.py)
![image](https://github.com/huggingface/transformers/assets/25120867/dbaf37e0-3a1a-4795-8613-4016332818bb)

2. a hard fix
After edit one filename of these two tokenizers, the bug disappear.

### Expected behavior

import the tokenizer with no exception
I think you should report the issue in the repositories where this code comes from, as the fix you suggest is in that code (and not in the Transformers library).<|||||>hmmm that's an ad-hoc solution to validate the problem, and I'm not sure whether this import error is expected by Transformers?<|||||>I don't see an import error in the code and traceback you pasted. I see the custom code of this tokenizer failing to execute.