Some more info here: https://github.com/pytorch/pytorch/issues/2517#issuecomment-325039259 43
I think this can be solved by setting mp.set_start_method('spawn') before any cuda call (including setting the rng, for example) when dealing with cuda share_memory objects.