having the same issue (DP much better validation metrics than DDP). setting
torch.backends.cudnn.enabled = False
slows my runtime down by 3x.
monkey patching torch.nn.functional.batch_norm
def monkey_patch_bn():
    # print(inspect.getsource(torch.nn.functional.batch_norm))
    def batch_norm(input, running_mean, running_var, weight=None, bias=None,
                   training=False, momentum=0.1, eps=1e-5):
        if training:
            size = input.size()
            size_prods = size[0]
            for i in range(len(size) - 2):
                size_prods *= size[i + 2]
            if size_prods == 1:
                raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))

        return torch.batch_norm(
            input, weight, bias, running_mean, running_var,
            training, momentum, eps, False
        )
    torch.nn.functional.batch_norm = batch_norm

doesn’t seem to help.
train_dataloader.sampler.set_epoch(epoch) doesn’t seem to help either.
EDIT:
what does seem to work is dividing my lr by my world_size, although i’m not sure why.