Solved by albanD in post #2 
              
              
                Hi, 
That depends how you implement your custom batch norm. 
Did you make sure to use .detach() properly when saving statistics not to keep backpropagating into previous batches.