Instead of using dataset.random_split I would create indices for both splits, create two separate datasets with the custom transformations etc. applied to the training and validation dataset, and wrap both datasets in Subsets using the indices.
This would allow you to pass the different arguments or transformations without manipulating some internal flags.