Solved by ptrblck in post #2 
              
              
                I get the same expected results if both approaches are using eval and training mode, respectively. 

During training the batch stats will be used to normalize the input (and the running stats will be updated) while the running stats will be used to normalize the input during eval. 
The docs explain …