Thanks for your direction. I have tried using torch.nn.parallel.DistributedDataParallelCPU and the forward pass is able to utilize the number of processes I set (I assume that’s the same as cpu cores in my case). I followed the tutorial here 968. However, there’s a lengthy block, for what I think is the backward pass, before any forward pass is observed.
Any suggestion on how to address this?