Solved by a_d in post #2 
              
              
                model.eval() changes the behaviour of a few layers like batchnorm and dropout. So if you want to infer using your model then put the model in .eval() mode or if you want to continue training then put the model into training using model.train(). 
Save at the least validation loss