Solved by ptrblck in post #3 
              
              
                You are comparing the native batchnorm layer in training mode with your FuseBN layer, which uses the eval logic. 
Also, after initializing the batchnorm layer the running mean would be all zeros and running_var all ones so you might want to train it for a few steps so that both layers would indeed n…