Hi,
I’m trying to use DistributedDataParallel on a CPU-only machine with multiple cores.
The documentation for DDP (https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py 37) states: "For multi-device modules and CPU modules, device_ids must be None or an empty list, and input data for the forward pass must be placed on the correct device. (default: all devices for single-device modules).
I want to parallelize training across CPU processes in a single machine. My dataset is an in-memory numpy array.
Would I have to manually separate this dataset into different subsets, and load each subset for each CPU process? Does splitting the input along the batch dimension work for CPU modules as well? I am using torch’s multiprocessing module to spawn processes to use with my DDP model.
Thank you.
PS. What’s the best practice for sharing an in-memory array across torch processes? That would be helpful as well.

Some additional observations:
When I print the loss in each process, the loss value is the same. If the data was being split properly by DDP, wouldn’t each process have a different loss value?