I have the following two variables in my dataset which is a list of image names per object category: self.list_image_names and self.num_images. The __getitem__ does something like this
image_names = [self.image_names[i][idx % self.num_images[i]] for i in range(len(self.list_image_names))]. The order in which my model sees the images changes due to the default sampler but the images inside a particular batch are the same every time (for a given idx). I plan to shuffle data at the end of every epoch using self.image_names[i] = np.random.permutation(self.image_names[i]) to break this correlation.
Implementing my own Sampler might be faster or the correct thing to do, but I think this shuffling will do what I want.