Hey @mrshenli
Thanks for tuning in, I understand why you are asking to measure the forward pass for both DP and DDP. But if you think about it from the end-user perspective, I care more about the overall training performance (which involves both model sync and data batches loading). If DDP provides better time only in the forward pass but (somehow) wastes the time it saved in data loading, then, as overall, DP will be a better option! Do not you agree?!