You could create a Dataset without any data augmentation, and another one with some augmentations.
Once both are created, you could use ConcatDataset to concatenate them into a single Dataset, which can be passed to a DataLoader.