Thanks for our answer. I understand that this is a solution. However, the point is that the 3D volumes are big and the patches relatively small. Doing the preprocessing and augmentations on the big volumes takes some time, therefore it is very inefficient to only sample one small patch per volume.
I solved this by wrapper this dataset in a UnbatchDataset (similar to the .unbatch() in tensorflow) and the BufferedShuffleDataset (only in pytorch nightly, part of the 1.8 release?).
However, now I encounter this issue after training for several hours: RuntimeError: unable to open shared memory object </torch_37358_1335134282> in read-write mode
Any ideas?