I’m sorry, perhaps those namings are not 100% accurate. Also, I’m sorry for your confusion. Perhaps this expression is easier to follow.
The problem is that, dataset distillation algorithm assumes that model weight (and running mean/var) stays unchanged during distillation (multiple running of forward  and  backward). But after a run of  forward  and  backward, the running mean/var is changed, leading to incorrect gradient computation in the next run of  forward  and  backward. Also, the number of synthesized images is very small (10-100 images), so we cannot simply use the statistics of synthesized images. The naive solution would be using batch norm always in eval mode ( track_running_stats=False ), which will harm model power. A more clever solution would be to track and add computation graphs for buffers, and then use gradient to update running mean/var, just like  synthesizing images in backward function.
And my question is, how to add running mean/var into computation graph while keep the bn layer behavior normal?