I have 2 batchnorm layers, both have same weight, same bias, same running_mean and same running_var. The first is a BatchNorm2D layer and second is a BatchNorm3D layer
I give an input x2d (1x128x64x64) to first BatchNorm2d layer and input x3d (1x128x5x64x64) to both layers. I ensure that
x3d[:,:,0,:,:] = x3d[:,:,1,:,:] = x3d[:,:,2,:,:] = x3d[:,:,3,:,:] = x3d[:,:,4,:,:]  = x2d

Thus they get same mean value (i.e.) :
(x2d.mean(dim=-1).mean(dim=-1)) == (x3d.mean(dim=-1).mean(dim=-1).mean(dim=-1))
I expected both batchnorm layers to give exact same output but it turns out that they were off by few percents!!
Any reason for this???  @ptrblck ?