As far as I understand you would like to switch between train and eval in every iteration in your training code?
So your code would look similar to this:
for epoch in range(10):
    for batch_idx, (data, target) in enumerate(train_loader):
        model.train()
        # your training code

        # Compute your stats in each itertation
        model.eval()
        stats = model(...)

If that’s the case, the code should run. I’m not sure, if you will introduce a lot of overhead to switch the modes in each iteration.
The running stats shouldn’t be reset. They will be updated in each forward pass, when your model is in training, and just applied if it’s in eval.