Hey @Martun_Karapetyan, DDP should have kept all model replicas in sync, i.e., all model replicas should have the same parameter values. Could you please check if this is true in your use case, say by using all_gather to collect all model parameters into one rank and compare?