Hello @ptrblck & @Yanli_Zhao, first thing first, thank you for messages, it helps !

Here’s what I changed :


dist.init_process_group(backend=args.backend) is done once the data-set is fully loaded. The objective is to get ride of potential sources of desynchronization.


I added a simple model (29 330 trainable parameters, 14x smaller than the original model). My goal is to reduce the communication overhead in distributed training.


I removed the call to the test function from the master’s training loop.



Here’s what I observe  :


Training times

To train the simple model with 1 GPU takes 47.328 WALL seconds
To train simple model with 3 GPUs takes 23.765 WALL seconds
To train the original model with 3 GPUs takes 26.433  WALL seconds
Training time is divided by two when I triple the GPU capacity. This looks like slightly better but still a bit unsatisfying.)



60% of the time is spent synchronizing (regardless of the model version: simple or original).

When training the simple model most of the time is spent synchronizing
When training the original model most of the time is spent in data transfert.
Based on the above, I guess that my original observation about time being spent in ncclKernel_AllReduce was wrong.



When training the simple model, the master is not spending time synchronizing. Would that indicates it is the slower node that others wait after ? or is that expected ? If this is expected, could the synchronizing time on node 1 and 2 be simply them waiting for the master doing the sum of all the weights ?



More details
For some reason until that point I did not noticed the “Distributed” view in the tensorboard “pytorch profiler” tool, it is quite insightful.

Simpler model

image3370×1276 151 KB


Original model

image3359×1281 154 KB


Profiler (simple model)
As @Yanli_Zhao suggested I loaded the profile in chrome but I’m not quite sure what I’m searching for. The dependencies of the ncclKernel_AllReduce_RING_LL_Sum_float(ncclWorkElem) do not look problematic to me (but I have no reference point of a working cluster to compare with). Below is a capture of chromium profiler.

image3770×1575 306 KB


Next steps ?

Setup and use the Nsight Systems to get a grasp on what’s going on ?
Anything else ?


The simple model definition is below.
class SimpleNet(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.fc = nn.Linear(20*12*12, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 20*12*12)
        x = self.fc(x)
        return F.log_softmax(x, dim=1)


ps. I ran few tests using simulated data, it does not indicate that my test throttles on data I/O (which is  expected since I force load the dataset in ram beforehand).