Hello,
I’d like to process my training dataset every epoch.
I want to add random processing as data augmentation, and I want to do it during training, not preprocessing.
I think I can do it with Trainer, DataCollator, or __getitem__ of datasets.arrow_dataset.Dataset, but where should I do it?
For the evaluation set and test set, I plan to do a preprocess using datasets.arrow_dataset.Dataset.map.
Thank you in advance.