The shuffling is only so that the batches that are seen by the model are different during every epoch. If you want to do cross-validation, you want to separate the data into a training and validation set before building the dataloaders.