oh yes, you are right! I should read the code more carefully.
For images, I found a ugly way to do this:
sample_idx = torch.randperm(len(dataset))[:sample_size]
dataset.samples = np.array(dataset.samples)[sample_idx.tolist()].tolist()

But your way is cleaner and more standard. Thanks!