I’m not sure, why the running stats updates are not gathered to the default device, but using
self.running_mean.copy_(...)
# instead of
self.running_mean = (...)

seem to perform the updates properly.
CC @SeoHyeong