my understanding was incorrect for the reason of the error.
According to https://github.com/pytorch/pytorch/issues/973 125 what is actually happening is that the multiprocessing Pool has a rotating set of workers, which are not waiting for the transfer of the file descriptors (of Tensor sharedmem) to finish from worker to main process, before which the workers are being recycled.
We’ll investigate this further on https://github.com/pytorch/pytorch/issues/973 125
I noticed that
pool = multiprocessing.Pool(multiprocessing.cpu_count(), maxtasksperchild=1

does fix the error (remove the set_start_method from the code, i’ll look into that separately). However, having only one task per child before it’s reinitialized is probably not very efficient.