Edit: Do not use this! I messed up. Because I did not specify the Dataset to be an IterableDataset, the iter method was not used. Instead it simply iterated over indices in the dataset instead of sampling random indices.
I still cannot find a way to make it efficiently run with a dataloader. Multiprocessing works, but is 8x slower with 2 workers and 16x slower with 8 workers.
@zer0ne @ptrblck I finally managed to get it working. One can use the standard Dataset (not the IterableDataset!) by implementing the methods:
def __getitem__(self, key):
    pass
    # return single transition: state, action, reward, next_state, done

and
    def __iter__(self):
        while True:
            idx = random.randint(0, len(self) - 1)
            yield self[idx]

Then we need to instantiate this class and create a dataloader based off it. The dataloader can then be used in this way:
    def sample(self):
        try:
            out = next(self.iter)
        except StopIteration:
            self.iter = iter(self.dataloader)
            out = next(self.iter)
        return out

This will work for any number of workers!
Of course a custom collate_fn  needs to be passed into the Dataloader creation.