Could you explain your use case a bit more as I’m not sure to understand it completely.
Currently you have three subnetworks using BatchNorm layers.
Would you like to change the update of the running averages, i.e. mean and std?
Or would you like to average the running averages of your three subnetworks somehow?