@dizvara DDP ensures that the gradients on all ranks are the same resulting in each rank having the same parameters after the optimizer step is run. However the output on each rank will be different and this is expected if you have different inputs on each rank. Looking at your code each rank does torch.randn(10, 10).cuda() for the inputs and as a result the inputs could be different resulting in different outputs on each rank.