I think the answer now is quite specific to the cluster and not quite relevant to PyTorch. At the high level, I think you need to write the output to a dedicated distributed/shared file system or directory instead of the local disk of each node. Better ask the cluster admins for the right output destination.