Okay, that’s pretty big, yes.
The caching idea was for a situation where memory constraint wasn’t the primary concern. And the custom sampler can help you match your random access patterns during train to the cache policy (how many files you can cache at the same time).
Is it safe to assume your numpy files are .npy and not an .npz? You could try the simplest approach and use the memory-map flag when loading your numpy files and see how the performance is. Depending on the combined size off all the numpy files, you may be able to load all the numpy file mappings once and leave them open.