That makes sense thanks! Although it’s a bit strange I didn’t come across anyone mentioning this problem on other forum posts. Seems important if you want to work with a frozen, pretrained CNN.
We DO want to keep the running mean and variances frozen after initialisation right? i.e. this is the correct way to use a pretrain Resnet? They should be set to desirable values from the Resnet’s pretraining?