It might not be the “correct” way to do it, but you could load #batchsize of images in the dataset and return all of them. This would mean that you tell the dataloader that you have a batch size of 1, but each sample in that batch has #batchsize images that you can reshape the way you want it.
If you use this method you could use the index in the get_item method in the dataset to load the class images of your choice.
Another way to do it is to write your own sampler as you suggested. I did something similar but it turned out to be a bit more code than I thought. If you decide to go this route and need some pointers, hit me up