}
Preprocess
The next step is to tokenize the text into a readable format by the model. It is important to load the same tokenizer a
model was trained with to ensure appropriately tokenized words. Load the DistilBERT tokenizer with the
[AutoTokenizer] because we will eventually train a classifier using a pretrained DistilBERT model:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
```
Now that you have instantiated a tokenizer, create a function that will tokenize the text. You should also truncate
longer sequences in the text to be no longer than the model's maximum input length:
python
def preprocess_function(examples):
    return tokenizer(examples["text"], truncation=True)
Use 🤗 Datasets map function to apply the preprocessing function to the entire dataset. You can also set
