The python and rust tokenizers have roughly the same API, but the rust tokenizers have a more complete feature set.
This introduces two breaking changes:
- The handling of overflowing tokens between the python and rust tokenizers is different.
- The rust tokenizers do not accept integers in the encoding methods.
How to obtain the same behavior as v3.x in v4.x
The pipelines now contain additional features out of the box. See the token-classification pipeline with the grouped_entities flag.
The auto-tokenizers now return rust tokenizers. In order to obtain the python tokenizers instead, the user may use the use_fast flag by setting it to False:
In version v3.x:
```py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
to obtain the same in version `v4.x`:py
