from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
```
2. SentencePiece is removed from the required dependencies
The requirement on the SentencePiece dependency has been lifted from the setup.py. This is done so that we may have a channel on anaconda cloud without relying on conda-forge. This means that the tokenizers that depend on the SentencePiece library will not be available with a standard transformers installation.
This includes the slow versions of:
- XLNetTokenizer
- AlbertTokenizer
- CamembertTokenizer
- MBartTokenizer
- PegasusTokenizer
- T5Tokenizer
- ReformerTokenizer
- XLMRobertaTokenizer
How to obtain the same behavior as v3.x in v4.x
In order to obtain the same behavior as version v3.x, you should install sentencepiece additionally:
