Solved by ptrblck in post #2 
              
              
                No, since the running estimates are buffers and do not require gradients, you don’t have to pass them to the optimizer. 


You can access them via my_bn_layer.running_mean and my_bn_layer.running_var.