Hey @Randool, as of PyTorch v1.9, DistributedDataParallel only supports Single-Program Single-Device mode, where each process exclusively works on a single model replica. In this case, DistributedDataParallel will not divide forward inputs, and instead will directly pass it to the wrapped local model. Will this be sufficient for your use case?