This code is confusing. Imagine if you forget to disable shuffle in one of the dataloaders or use multiple workers.
Simpler way here is to define your custom dataset with all the data. Also you can move all preprocessing outside of the loop either with dataset’s init/get_item or with collate_fn.