Simple way to convert a Python tokenizer to a fast tokenizer
# 🚀 Feature request

Tokenizer are provided with each model, some have a fast version of their tokenizer (Rust based), others like CamemBERT have only the slow version.

## Motivation

Fast tokenizer improves inference times drastically (in real time inference for instance).
Plus there is no reason it should not be possible

## Your contribution

If you provide me with basic guidelines on how to manually make a conversion, I can submit a PR to offer such feature.

requires unigram algo implemented on tokenizers
https://github.com/huggingface/tokenizers/pull/292<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>PR merged, closing the issue<|||||>Hi @pommedeterresautee, could you please refer me to how I can convert an existing Python tokenizer to a Fast tokenizer?

Sorry if I missed something, and thanks so much for your help!<|||||>Maybe @SaulLu or @Narsil can comment and link to an example!<|||||>Hi @varun-tandon ,

The code to change from slow to fast is included here: https://github.com/huggingface/transformers/blob/main/src/transformers/convert_slow_tokenizer.py

As you can see there are many variables, depending on the actual model and what you want to achieve.

Usually it involves understanding how a model actually does the tokenization (and all the bits like CLS, SEP etc..) and using the compponents of `tokenizers` to assemble them to make the output similar to what the python code does:
https://huggingface.co/docs/tokenizers/components

Sometimes we're missing a brick and we simply add it (although it becomes rarer with time)
