To solve “RuntimeError: the derivative for ‘running_mean’ is not implemented”, you need to make sure that batchnorm_mod.running_mean.requires_grad == False.
What might be happening is that in your loading script, you treat both Parameters and buffers the same way, and so you make all of the require gradients. But you want to be careful and make sure that you don’t set requires_grad=True for the buffers.