Hi,
I was wondering if there is a significant disadvantage placing a dataloader and dict inside a for loop.
I am trying something where a random sample of training data is used for each epoch as I have millions of examples but not enough resources for effective training. Therefore, I want to use a bash script that randomly copies a certain amount of images from several directories and use this for training. The same is also applied for validation.
The pseudocode would look something like this:

for epoch in epochs:

     run bash script to copy random number of train and validation data per epoch
     
     create dataloader dicts and dataloader
    
     do training
     do validation


    delete train and validation data 


Would this be ok?