@Mahmoud_Abdelkhalek  Just for fun I am exploring this problem myself.
Got it. The problem is that you are reusing the same batch norm layer multiple times. When you copy thebatch norm value, it’s gonna change the values to the previous batch norm modules too (because they are the same module!).
The problem is here:
stages = [nn.Sequential(conv1,
                        batch_norm,
                        activation,
                        conv2,
                        batch_norm,
                        activation,
                        pooling)]

Use different batch norm layers and you problem will be solved.