Hi @sami, thank you very much for your answer. I understand what you did above, and even though I don’t want to use it in a convolutional (or any other parametric operation), your approach should do the job for me. But it doen’t…
Let’s say that you have a 4-dimensional tensor X of size (bs, ch, d, d):
bs = 2, ch=3, d=4
X = torch.randn(bs, ch, d, d)

Then, Y = bn(X) will forward X through the BN layer, by computing bn.running_mean, bn.running_var, etc…
I would like to reproduce the value of Y by manually applying the operation of BN. That should be the following:
Y_ = bn.weight.reshape(1, ch, 1, 1) * (X - bn.running_mean.weight.reshape(1, ch, 1, 1))/torch.sqrt(bn.running_var.weight.reshape(1, ch, 1, 1) + bn.eps) + bn.bias.weight.reshape(1, ch, 1, 1)

But it doesn’t seem to work. I tried also with a BN layer without any affine transformation, but it still cannot reproduce Y.