nima_rafiee:

why should I make a new_group() and


Hi, thanks for posting your solution. It perfectly solves my problem.
But I’m still wondering why our situations, DDP doesn’t work with normal BN.
Both BN and SyncBN are supported by DDP and work well in my toy model:
class ToyModel(nn.Module):

    def __init__(self):

        super(ToyModel, self).__init__()

        self.net1 = nn.Linear(10, 10)

        self.relu = nn.ReLU(inplace=True)

        self.bn = nn.BatchNorm1d(10)

        self.net2 = nn.Linear(10, 5)

    def forward(self, x):

        return self.net2(self.bn(self.relu(self.net1(x))))


Found the cause in my case: Inplace error if DistributedDataParallel module that contains a buffer is called twice · Issue #22095 · pytorch/pytorch · GitHub 24