Your workflow should generally be alright, but I would be concerned about the performance hit caused by the copies of the data. If possible, leave all data in the current structure and use a Subset or SubsetRandomSampler to load only the desired samples instead.