Are you sure that memory usage is the most serious overhead ? What about IO usage ?
Setting too many workers might cause seriously high IO usage which can become very uneffective.
I would love to get your advice about the recommended way to deal with my data - I feed my CNN with large batches (256/512/1024…) of small patches of size 50x50. I intend to use the ImageFolder DataLoader for that, but I’m afraid that it would be very uneffective to load from disk a lot of small images in high frequency.