Is there a way to use the runing mean and varaince of batch norm layer during training? I want to finetune a model without using the batch mean and variance, the reason is that there is no easy way to synchronize batch mean and variance between several GPUs, and the minibatch size is so small in each GPU.
Thank you.