Thank you for your reply.

Does this mean each DDP process is consuming 4096 samples per iteration and the DP process is consuming 4096 * 8 = 32768 samples?

No, I’m talking about the global batch size, which means a DDP process consumes 512 samples per iteration.

BTW, how did you initialize DDP? Could you please share a repro?

OK, I will try to provide a short version that can reproduce the performance shortly, since the original program is very long because of automation and visualization. Thank you.