Thank you @zazzyy very much for your kind and detailed answer. It’s really helpful.
Maybe I haven’t well-explained in my first post. In fact, what I would like to do here is freezing  several modules of a pre-trained model and pass other modules (including a resnet block with batch norm layer) to optimizers to train. However, when I checked the parameters passed to optimizer, I found that  running_mean and running_average are not included.
That’s why I’am wondering if the running_mean and running_average could be updated during the training.
I supposed that pytorch will still update the running_mean and running_average even though they are not explicitly passed to optimizers.