Hi everyone,
I’m trying to understand what is the difference between the DistilBertTokenizerFast 18 and the DistilBertTokenizer 3.
From the documentation it looks like that the DistilBertTokenizerFast “Construct a “fast” DistilBERT tokenizer (backed by HuggingFace’s tokenizers library).” whereas the DistilBertTokenizer “Construct a DistilBERT tokenizer.”, but I don’t understand what that means.
This is what I found from the documentation: Tokenizer — transformers 4.5.0.dev0 documentation 8
To the best of my understanding the difference between a “fast” and a “non-fast” tokenizer is computation speed but there is no functional difference between them. Please correct me if I’m not the right direction.
Any help will be greatly appreciated.
Thank you,
Ayala