from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
```
As we saw in preprocessing, we can prepare the text inputs for the model with the following command (this is an
example, not a command you can execute):
python
inputs = tokenizer(sentences, padding="max_length", truncation=True)
This will make all the samples have the maximum length the model can accept (here 512), either by padding or truncating
them.
However, we can instead apply these preprocessing steps to all the splits of our dataset at once by using the
map method:
```python
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
```
You can learn more about the map method or the other ways to preprocess the data in the 🤗 Datasets documentation.
