Hi, DDP will broadcast model’s weights to all GPUs when you construct DDP. DDP will reduce gradient when you call backward(). DDP takes care of broadcast and all_reduce so that you can treat them as if they are on a single GPU (This is only true when you do standard network training that updates model after one forward-backward pair).