You need to set the batchnorm modules to eval() mode. If you are calling batchnorm with the Functional interface, this involves setting “training=false.” Otherwise just make sure all the modules you aren’t training have had .eval() called on them. Note that .train() or .eval() calls propagate to all child modules, so if you call .train() on your whole network you’ll have to call .eval() separately on the modules you don’t want to finetune.