Hi,
In a specific application, I need to freeze running statistic calculation in BatchNorm layer in a part of my code, but I need to utilize “gamma(weight)” and “beta(bias)” of this layer in training with gradient forward\backward pass.
I have implemented this by building an extra BatchNorm layer with “affine” false and doing forward pass as:
base_BatchNorm = nn.BatchNorm1d(1200)
extra_BatchNorm = nn.BatchNorm1d(1200, affine=False)
x = extra_BatchNorm(x)*base_BatchNorm.weight+base_BatchNorm.bias
I am curiose if there is more clean way to implement this. I think this can be done by setting momentum=0? or by a combination of  momentum=0 and affine=False for base_BatchNorm  as:
base_BatchNorm.momentum = 0
base_BatchNorm.affine = false
x = base_BatchNorm(x)
base_BatchNorm.momentum = 0.1
base_BatchNorm.affine = true
Is it right?