Yes they are in the same process group and yes I am using DistributedDataParallel
Here is a quick code snippet and output
    if self.id == 0:
        ep = 10
    elif self.id == 1:
        ep = 20

    for i in range(0, ep):

        self._log.info(self.id, "image.. " + str(i))
        image = torch.zeros((1,2,3,256,384)).cuda(self.id)
        input = {"rgb": image}
        self._log.info(self.id, "model.. " + str(i))
        output = self.model(input)
        loss = torch.sum(output["output"][-1])
        time.sleep(0.1)

        self._log.info(self.id, "zerograd.. " + str(i))
        self.optimizer.zero_grad()
        self._log.info(self.id, "backward.. " + str(i))
        loss.backward()
        self._log.info(self.id, "step.. " + str(i))
        self.optimizer.step()
        #print("step: " + str(self.id) + " " + str(i))
        self._log.info(self.id, "done " + str(i))

    self._log.info(self.id, "barrier.. ")
    dist.barrier()

Here, I am having two processes run. And one finishes faster than the other and needs to wait on the first one. But instead as soon as 0 calls barrier, it blocks 1 completely.
Here is the output
[INFO]  [0] zerograd.. 9
[INFO]  [0] backward.. 9
[INFO]  [0] step.. 9
[INFO]  [1] model.. 10
[INFO]  [0] done 9
[INFO]  [0] barrier.. 
[INFO]  [1] zerograd.. 10
[INFO]  [1] backward.. 10
[INFO]  [1] step.. 10
[INFO]  [1] done 10
[INFO]  [1] image.. 11

As you can see process 1 is stuck on image…