yes, for the synch part, that’s what i was doing (model is self synch using ddp, but for other objects i use torch.distrbuted.all_gather). so, this is fine.
for writing, i allow only node master to write.
for the last point, this is fine on paper but it does not work… not because ddp or pytorch but because of slurm.
i ask every master node to write/bring/decompress some data from somewhere else to the node local disk. but at the end, they did all instruction, but processes dont find decompressed files. i wrote to the server it staff asking for some explanations. waiting…
so, i’ll face the same issue when writing checkpoints, or any file in each process… later when asked to reload the checkpoint, it will be missing.
thanks