token to index (that we usually call a vocab) as during pretraining.
To automatically download the vocab used during pretraining or fine-tuning a given model, you can use the
[AutoTokenizer.from_pretrained] method:
```py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
```
Base use
A [PreTrainedTokenizer] has many methods, but the only one you need to remember for preprocessing
is its __call__: you just need to feed your sentence to your tokenizer object.
```py
encoded_input = tokenizer("Hello, I'm a single sentence!")
print(encoded_input)
{'input_ids': [101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102], 
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
