Solved by ptrblck in post #8 
              
              
                The mean and variance are not hardcoded. Both will be initialized and updated during training, i.e. the current batch will be normalized using its mean and variance, so that mean=0 and var=1, while the running_mean and running_var will be updated using the momentum term. During evaluation (calling m…