Solved by ptrblck in post #4 
              
              
                The model.train() and model.eval() calls will switch between the training mode (normalizing the input batch with its own stats and updating the running stats) and evaluation mode (normalizing the input batch/sample with the running stats). You don’t need to apply the running stats manually.