Hey @yllgl, this might be caused by a coalesced broadcast on buffers. Does your model contain buffers? If so, do you need DDP to sync buffers for you? If not, you can try setting broadcast_buffers=False in DDP ctor.