Thanks for your reply.
I have tried compare BN in checkpoint and original BN.
It’s true that the grad is correct but running_mean and var were updated twice.
I can’t figure out why the grads won’t be wrong.