My workaround was like:
# indices to draw samples from the dataset.
picks = np.random.permutation(20)

dataloader_train = DataLoader(
    dataset,
    batch_size=batch_size,
    shuffle=False, # note that sampler and shuffle arguments are mutually exclusive
    sampler=picks,
    collate_fn=dataset.collate_fn
)