Since ImageFolder will lazily load the data in its __getitem__ method, you could create three different dataset instances for training, validation, and test and could pass the appropriate transformation to them.
You could then create the sample indices via torch.arange(nb_samples) (or the numpy equivalent) and either split these indices manually or with e.g. a sklearn method, which also allows you to apply stratified splits etc.
These indices can then be passed together with a dataset to a Subset instance to create the final datasets.
The passed indices will be used to load the samples, and since you won’t load any data beforehand, you won’t waste memory.
Alternatively, you could also pass the indices to RandomSubsetSampler and pass the datasets to a DataLoader instance.
Let me know, if that would work.