solarflarefx:

If NCCL is supported only in Linux and DDP uses NCCL, how would the support in Windows actually work?


Currently, DDP can only run with GLOO backend.

For example, I was training a network using detectron2 and it looks like the parallelization built in uses DDP and only works in Linux.

MSFT helped us enabled DDP on Windows in PyTorch v1.7. Currently, the support only covers file store (for rendezvous) and GLOO backend. So when calling init_process_group on windows, the backend must be gloo, and init_method must be file. To run on a distributed environment, you can provide a file on a network file system. Please see this tutorial 31 and search for win32 .

Would the speeds be slower as compared to doing the same training on Linux?

Yep, Gloo backend will be slower than NCCL.

Finally, how would using WSL2 compare to the native Windows support of DDP?

I am not aware of this. Let me ask MSFT experts to join the discussion.