I assume “trouble in memory management” means you are running out of host RAM?
If so, make sure that loading the entire (small) dataset fits into the RAM as well as a batch using the larger images (intermediates could also be stored). In case you are using multiple workers, note that each worker would copy the Dataset and thus also the preloaded small dataset.
If you are using the CPU for the actual training then note that also the model training (parameters, forward activations, gradients etc.) also need to be stored in the RAM.