Solved by ptrblck in post #2 
              
              
                nn.DataParallel would update the batchnorm stats on the default device, as seen in this code snippet:

bn = nn.BatchNorm2d(3, momentum=1.0).cuda()

a = torch.randn(16, 3, 224, 224) * 12 + 7
b = torch.randn(16, 3, 224, 224) * 5 - 9
x = torch.cat((a, b), dim=0)

model = nn.DataParallel(bn, device_id…