I am trying to use an update scheme for the moving average of the batch normalization in my model. Does anyone have an idea how to achive this?
Here is a snippet of my network:
for subnetwork in ['h_net', 'o_net', 'c_net']:
    setattr(self, subnetwork, nn.Sequential(nn.Linear(input_dim, 80),
                                            nn.ReLU(),
                                            nn.Batchnorm1d(80),
                                            nn.Linear(80, 40),
                                            nn.ReLU(),
                                            nn.BatchNorm(40),
                                            nn.Linear(40, 1),
                                            nn.ReLU()))

In the end I sum up the results of the different subnetworks.