Arun_Vishwanathan:

The issue is basically the current batch is used to make the output of bn in eval mode.


Could you check the output again?
Using your model and this code snippet:
torch.manual_seed(0)
net = Net()
net.bn.bias.requires_grad=False
net.bn.weight.requires_grad=False
with torch.no_grad():
    net.bn.weight.fill_(1.)
net.bn.eval()

x = torch.randn(1, 3, 5, 5)
net(x)

yields the exactly same outputs:
conv output
tensor([[[[ 0.4132035971, -0.7129006982,  0.0829921290],
          [ 0.5575969219, -0.5301585793, -0.2478107512],
          [ 0.2830447555, -0.8774284720, -0.7162327170]],

         [[-0.1680016071, -0.6251038313, -0.3634709716],
          [ 0.5589009523,  0.4439742863,  0.1032543331],
          [ 0.4814728498,  0.3313015699, -0.5421049595]]]],
       grad_fn=<MkldnnConvolutionBackward>)
bn output
tensor([[[[ 0.4132015407, -0.7128971219,  0.0829917118],
          [ 0.5575941205, -0.5301558971, -0.2478095144],
          [ 0.2830433249, -0.8774240613, -0.7162291408]],

         [[-0.1680007726, -0.6251006722, -0.3634691536],
          [ 0.5588981509,  0.4439720511,  0.1032538190],
          [ 0.4814704359,  0.3312999010, -0.5421022177]]]],
       grad_fn=<NativeBatchNormBackward>)
...