Hey @111344, if you are looking for mathematical equivalence, you will need at least two things:

each DDP process processes batch_size / num_gpu samples: this allows DDP collectively process the same amount of inputs as local training.

loss_fn(model([sample1, sample2])) == (loss_fn(model([sample1])) + loss_fn(model[sample2])) /2: this is because DDP uses AllReduce to compute the average gradients across all processes. If the above condition is not met, then average gradients across all DDP processes are not equivalent the the local training gradients.

However, practically, applications usually do not have to satisfy the above conditions. Did you see any training accuracy degradation when scale up to 3 GPUs?