Hi Shen, could you please also take a look at my latest post which is relevant to your tutorial: DistributedDataParallel: model weights and grads not synchronized with multiple forward backward pass 8? Thanks!