Hi,
I have searched for the answer for my question, but still can’t get the clear answer.
Some issues in the github/forum also report that the result of tokenizer and tokenizerfast is a little bit different.
I want to know what is the difference between them (in terms of mechanism)?
If they should output the same result, then why we need both of them?