Have a look at Detectron2’s RepeatFactorTrainingSampler 17, for an example of one approach. Even if you don’t use a Detectron2 model, the sampler is just standard Pytorch code and should work fine with other models.  I found that it didn’t deal properly with empty images, however, and I modified it a bit for my own use.