Solved by ptrblck in post #2 
              
              
                You could reuse the same batchnorm layer, but this would indeed reuse the same affine parameters as well as update the running stats with each input, so it’s most likely not what you want. 
The standard approach would be to create a new batchnorm layer.