Do you mean in terms of the time it takes to run the experiment or in terms of accuracy when you mention performance?
The issue is I am worried if it means loading an entire dataset into a dataloader or dict before the random sampling. Is this something I should be concerned about or does random sampler not even load the entire dataset and then select random samples?
The issue is my dataset is also incredibly imbalanced. I want to copy all of the images from one class but from the other class, I would like to select random samples.  There is a further abstraction where there are small patch images selected from entire images. I want to select a certain number of these patches from each entire image. The directory looks something like this:

class 1 --- image 1 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

             --- image 2 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

                --- image ... --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...


class 2  --- image 1 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

             --- image 2 --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...

                --- image ... --- patch 1
                               --- patch 1
                               --- patch 3
                                --- patch ...