I’m not sure how you are saving the DataLoaders. Are you storing the Datasets including all loaded samples?
If so, the approach should be working fine. Alternatively, you could of course also lazily load the data and use a single Dataset.