There’s the learned weight and bias, but also a separate running mean and running variance that it keeps track of outside of any optimizer. If you want these not to update you need to set the BatchNorm module’s momentum to 0.