Next we will generate a small subset of the training and validation set, to enable faster training:
python
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
full_train_dataset = tokenized_datasets["train"]
full_eval_dataset = tokenized_datasets["test"]
In all the examples below, we will always use small_train_dataset and small_eval_dataset. Just replace
them by their full equivalent to train or evaluate on the full dataset.
Fine-tuning in PyTorch with the Trainer API
Since PyTorch does not provide a training loop, the 🤗 Transformers library provides a [Trainer]
API that is optimized for 🤗 Transformers models, with a wide range of training options and with built-in features like
