import torch
from torch import nn
from torch.autograd import Variable

torch.manual_seed(1)

net = nn.Sequential(
    nn.Linear(100, 100),
    nn.BatchNorm1d(100),
    nn.Linear(100, 1)
)

im = Variable(torch.ones(10, 100))
for _ in range(10):
    net.train(True)
    print(net(im))
for _ in range(10):
    net.train(False)
    print(net(im))

the first 10 prints output arrays with one value(-8.6313 * 1e-2), and the second 10 prints output arrays with another value (-0.1563 * 1e-2).
It is rather strange and confusing.


the first 10 output is the same, is it reasonable? I think the runing_mean and running_var have some initial value and then get updated at each step. so the output should be changing.


the second 10 output is the same, it is reasonable since it’s in eval mode. what is confusing is that it is not the same as the last training output.  I think if we set the model to evaluation mode, it just freezes runing_mean and running_var, which should make the second 10 outputs the same as the last training output.


Any ideas? I’m almost driven crazy by this!