Solved by mfayyaz in post #17 
              
              
                The problem is that your BN layers differ. 
I used the following code to solve the problem (just override the train function of your model): 
    def train(self, mode=True, freeze_bn=False, freeze_bn_affine=False):
        
        super(MyModel, self).train(mode)
        if freeze_bn:
            f…