BatchNorm2d works even when batch size is 1, which puzzles me. So what is it doing when batch size is 1? The only related thread I could find is https://github.com/pytorch/pytorch/issues/1381 57 without much explanation.
minimal example:
x = torch.FloatTensor(1,1,2,2)
x[0,0,0,:] = 1
x[0,0,1,:] = 2
m = nn.BatchNorm2d(1)
y = m(Variable(x))

// output is
// (0,0,.,.) =  [[-0.1544, -0.1544], [0.1544, 0.1544]]