I made a mistake above when launching distributed training with torch.distributed.launch I should have specified --nproc_pre_node.
So I did that running torch.distributed.launch --nproc_per_node=2 where 2 corresponds to num of process for each gpu in the system, and the launch help manual recommends that but I got even worse results not faster.


Also this message

image1894×70 23.8 KB

Unless I’m missing something fundamental here why is it said that distributed parallel is faster than data parallel?