Solved by ptrblck in post #2 
              
              
                model2's parameters won’t be updated unless you call model2_opt.step(). 
However, if your model contains batchnorm layers, the running estimates will be updated during the forward pass, if you keep these layers in .train() mode. 
Also, dropout will be applied, but of course there is nothing to updat…