I was trying to save the copied layers in a dictionary. Then on forward I would iterate through the modules and assign the right copy to the right bn layer. But I dont know how to do that other than explicitly assigning to each layer.
However I managed to save the batchnorm parameters instead of the whole layers in two OrderedDict(), one for real and one for synth. So in the beggining of each epoch I can simply load_state_dict(strict=False) to load only the parameters contained on the dict. It seemed to work, but I am not sure if this is efficient…