makslevental:

I split my network across the layer boundary so yes each DDP now sees a different model


I see. Is this forward only for inference or do you also run backward for training? If it is the latter, this might kill the correctness of DDP, as DDP expects the model in each process to be exactly the same, otherwise, the AllReduce communication across DDP process could mess up the gradients.

why is there cross-talk? I with  with torch.no_grad()  and  for p in model.parameters(): p.requires_grad = False  in every  run .

Looks like you are doing inference instead of training? In this case, don’t you need to somehow gather/combine the outputs from the four different Conv2d layers from 4 different DDP processes? Otherwise, how did you get the final inference result?
BTW, since this is inference only, why do you need DDP?