Solved by mariosasko in post #4 
              
              
                If you update them by reassigning the value like this self.running_means[i] = torch.tensor(...), the buffers won’t get updated, but if you change these tensors in-place then it should work, e.g. self.running_means[i].add_(...).