I see.
I think I can answer at least this question now once and for all for the community: Yes, parameters are updated in the forward pass. Even if requires_grad=False or volatile=True (which is a good thing since you can save memory then).
Here is the proof:
>>> net = nn.Sequential()
>>> net.add_module("conv",nn.Conv1d(1,1,1))
>>> net.add_module("bn",nn.BatchNorm1d(1))
>>> net[1].running_mean
0
>>> net(Variable(torch.ones(1,1,1)),volatile=True)
>>> net[1].running_mean
-0.1036

However, I still wonder if this is the correct way. As far as I can see, BN uses a fixed momentum, thus it is not invariant to the order of the sequence.
But I think for most datasets and if you use proper randomization, this should be a good enough approximation.
I think if you wanted to do this properly, you’d really have to calculate the layer-wise statistics by yourself using a fair average over your dataset and then set the buffers manually, which involves a lot of coding. For this, a build-in functionality would be really great (like fit_batchnorm_params_to_dataset or similar)