@googlebot Thanks for your response.
I do not quite understand you here:

Batch normalization uses buffers for running_mean, running_var, i.e. they’re updated without autograd.

running_mean and running_var are used in calculating the layer output, and for this reason backward() needs their gradients.
As for my purpose, I want to use Exponential Moving Averages in my loss function similarly to the way they are used to calculate the layer outputs in Batch normalization.