You could create a custom Subset or SubsetRandomSampler for each training “iteration”.
I.e. in the first iteration you would select 100 random indices and store the desired 20 indices after the training is done. Afterwards you would recreate the Subset or sampler as well as the DataLoader and repeat this workflow.