The traditional way of doing it is: passing an additional argument to the custom dataset class (e.g. transform=False) and setting it to True` only for the training dataset.
Then in the code, add a check if self.transform is True:, and then perform the augmentation as you currently do!