ad26kr:

However, I am still curious about the reason to separate them, since I found them very different in computation speed. Why not just use the fast one?


oh that’s because we do not have rust implementations + python bindings for every type of tokenizer that’s released by the various research groups. by default transformers will look for the fast implementation if it exists, or fall back to the “slow” one when it doesn’t