If you set num_workers > 0, each worker will load a batch in the background and inplace modifications of your Dataset won’t be noticed until the next epoch.
As you said, num_workers=0 might work and I can’t see any reason to use multiple workers, as your workflow seems to be sequential, i.e. you can just sample the next batch after do finishes.
So even if you would use multiple workers, all would have to wait until your loop finishes.