Solved by tom in post #2 
              
              
                There are three things to batchnorm 

(Optional) Parameters (weight and bias aka scale and location aka gamma and beta) that behave like those of a linear layer except they are per-channel. Those are trained using gradient descent and by disabling gradients you inhibit that they are updated.
There a…