I think your lazy loading approach is fine to use in your actual training pipeline.
However, you would need to grab all target values once to set up the sampler.
If you’ve stored to weights, you could then just load them in your actual training and could stick to your Dataset implementation. Let me know, if this would work.