I fix the bug strangely by no longer restoring the running_mean and running_var for backward.
The in-place operation like self.running_mean.copy_(self.prev_running_mean) when backward cannot pass the nn.DataParallel case.