@mrshenli I split my network across the layer boundary so yes each DDP now sees a different model - imagine instead of a 48 output channel Conv2d applying 4x12 output channel Conv2ds.



 mrshenli:

Since with the split, each forward pass will do cross-device communications, so it is possible to see slowdowns.


why is there cross-talk? I with with torch.no_grad() and for p in model.parameters(): p.requires_grad = False in every run.
.