May I ask, how you are shuffling the data in the Dataset?
Since the __getitem__ function receives an index, I assume you are shuffling by copying your data?
Maybe you could implement your own Sampler 21, which could be faster?