How to achive character lvl tokenization? (cant convert from huggingface/tokenizers)
Initially, I thought that huggingface/tokenizers is the same thing as tokenization in this repo.
I made it like this:
```
from tokenizers import Tokenizer, models, pre_tokenizers
from tokenizers.processors import TemplateProcessing

tokenizer = Tokenizer(models.WordLevel(unk_token='[UNK]'))
tokenizer.pre_tokenizer = pre_tokenizers.Split("", "isolated")


trainer = tokenizer.model.get_trainer()
trainer.vocab_size = 100
trainer.special_tokens = ["[UNK]", "[PAD]", "[SOS]", "[SEP]", "[EOS]"]

tokenizer.train(files=["alchemist.txt"], trainer=trainer)
tokenizer.post_processor = TemplateProcessing(
    single="[SOS] $A [EOS]",
    pair="[SOS] $A [SEP] $B:1 [EOS]:1",
    special_tokens=[
        ("[SOS]", 2),
        ("[SEP]", 3),
        ("[EOS]", 4),
    ],
)
tokenizer.enable_padding(pad_id=1, pad_token="[PAD]")
```

Still, huggingface/tokenizers lack some features I want like returning tensors.
So I tried to convert it to transformers tokenizer as suggested here
https://github.com/huggingface/tokenizers/issues/669#issuecomment-828864108

But got error:
```
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1631                 FutureWarning,
   1632             )
-> 1633             file_id = list(cls.vocab_files_names.keys())[0]
   1634             vocab_files[file_id] = pretrained_model_name_or_path
   1635         else:

IndexError: list index out of range
```

So, is tokenizer from tokenizers + converting to transformer tokenizer is a best way to achieve what I want?
If so, how should I convert it?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I just can not figure out why two api can not be the same. Extra Burden for users