I think the short answer is that the the fast (Rust-based) tokenizers are newer and will at some point completely replace the Python-based tokenizers. They are much faster and I’m not sure if there’s a good reason not to use them at this point, but I’m not the expert there. cc @mfuntowicz