Solved by isalirezag in post #2 
              
              
                I was dealing with that ***** the whole day, finally i think i got it, adding this will make BN not trainable: 

        def set_bn_eval(m):
            classname = m.__class__.__name__
            if classname.find('BatchNorm2d') != -1:
              m.eval()
    
        model.apply(set_bn_eval)