[Question]: Register new Tokenizer
Hi there,

I'm in the process of creating a new Transformer model. I have my own codebase and I'm using Transformers as an external library. If I implement a new Tokenizer that inherits from an existing one (say the BERT one) is there any way to "register" my new tokenizer so that Huggingface automatically instantiate it? I would like to support the `AutoTokenizer` API:

```python
tokenizer = AutoTokenizer.from_pretrained("heriot-watt/my_model_name")
```
And I would like that `AutoTokenizer` looks in my PYTHONPATH and automatically resolves the `Tokenizer` class with the name `my_model_name`. I've seen that currently, Transformers uses a hardcoded resolution strategy defined in `configuration_auto.py` or `tokenization_auto.py`. For instance, AllenNLP uses a nice register annotation to automatically resolve models, dataset reader and so on. What would be the best solution here? 

Thanks for your answer,
Alessandro
Hi! `AutoTokenizer` is only used to redirect to the correct tokenizer implementation under the hood, and not to resolve to any tokenizer object. The procedure here would be to create your tokenizer like you want it to be, either by using the `tokenizers` library, by tweaking an existing one or by creating yours from scratch.

Then, you can open a PR on the repo and have your tokenizer/model be added to the available architectures, and available in the `Auto*` classes so that others may leverage your checkpoints easily.<|||||>So I take you're not planning to have an automatic module discovery. I see. Anyway, I feel like an equally nice way to solve this is to have a folder on your current path called `heriot-watt/my_model_name`. In it, I have my config files and tokenizer files that belong to the `Tokenizer` I'm inheriting from. Then, In my package `__init__.py` I had to add the following:
```python
MODEL_MAPPING.update({
    MyModelConfig: MyModel
})

CONFIG_MAPPING.update({
    "my_model": MyModelConfig
})

TOKENIZER_MAPPING.update({
    MyModelConfig: (MyModelTokenizer, MyModelTokenizerFast)
})

MODEL_NAMES_MAPPING.update({
    "my_model_name": "MyModel"
})
```
In this way, I'm able to use the `Auto*` API just fine :) <|||||>Thanks for showing us how you do it! That's a very interesting usage of the AutoModels, and definitely something we would be interested in adding. For example via a `transformers.register_auto_model(xxx)` or something along those lines.<|||||>I think the AllenNLP registrable is a very good starting point for this: https://github.com/allenai/allennlp/blob/main/allennlp/common/registrable.py<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Maybe this is reckless, but I could see value in at least partially inverting this relationship.  If my `.save_pretrained()` implementation could drop a hint about what module an implementation resides in, Auto Classes could have the ability to try a dynamic import without needing any registration api, and the `Auto*.from_pretrained()` caller would be relieved of the burden of making sure implementation classes are loaded ahead of time.

I honestly went looking for where this happened in the code multiple times and assumed I just hadn't figured out how it worked yet.<|||||>This is sloppy and hardly thought through, but
```diff
diff --git a/src/transformers/models/auto/tokenization_auto.py b/src/transformers/models/auto/tokenization_auto.py
index f07e366c7..3ad9d1e22 100644
--- a/src/transformers/models/auto/tokenization_auto.py
+++ b/src/transformers/models/auto/tokenization_auto.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 """ Auto Tokenizer class. """

+import importlib
 import json
 import os
 from collections import OrderedDict
@@ -538,6 +539,10 @@ class AutoTokenizer:
             if tokenizer_class is None:
                 tokenizer_class_candidate = config_tokenizer_class
                 tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
+            if tokenizer_class is None:
+                tokenizer_module = tokenizer_config.get("tokenizer_module")
+                tokenizer_module = importlib.import_module(tokenizer_module)
+                tokenizer_class = getattr(tokenizer_module, config_tokenizer_class)

             if tokenizer_class is None:
                 raise ValueError(
```
for example, would allow subclasses that were not officially included with `transformers` to use
`super().__init__(..., tokenizer_module=self.__module__, ...)` in their constructor.  That seems to be enough for the setting to save in the tokenizer_config.json file.  Then the caller would no longer have to be aware of what imports are necessary for a `.from_pretrained()` call to succeed.<|||||>After 9870093f7b31bf774fe6bdfeed5e08f0d4649b07 I am unsure how to use a third party tokenizer class because `transformers.models.auto.tokenization_auto.tokenizer_class_from_name()` is using
```python
module = importlib.import_module(f".{module_name}", "transformers.models")
```
and trying to load and trying to use anything outside of transformers raises
```
ValueError: attempted relative import beyond top-level package
```
The workaround I have at the moment is adding 
```python
transformers.models.auto.tokenization_auto.TOKENIZER_MAPPING_NAMES.update((
    ("MyModel", ('MyModelTokenizer', 'MyModelTokenizerFast')),
))
sys.modules['transformers.models.MyModel'] = sys.modules[__name__]
```
to replace the `TOKENIZER_MAPPING` patch used in previous versions.  But dynamically patching in additional modules seems far more aggressive than updating data structures.

It would have been very convenient here if the module names in TOKENIZER_MAPPING_NAMES had included the "." rather than it being added by `tokenizer_class_from_name()`.