I’m trying to figure out DistributedDataParallel (on a single machine; single GPU / process mode). I’ve got a few questions:


Will all launched processes do CUDA init? Is it safe? Should we explicitly set CUDA_VISIBLE_DEVICES per launched process to ensure that it can see only one device? ( think fork/spawn global state issues with CUDA / OpenMP / pthreads etc…) Would it prevent it from rpc using nccl?


Is it sensible to do a forced parameter sync once in a while? Inherent GPU parallelism non-determinism can cause parameter divergence that could cause replica parameter divergence for some sensitive models (even if gradients are sychronized)? How does one do it?


When does it make sense to do NUMA node pinning? CPU affinity pinning?


Thank you!