Solved by ptrblck in post #3 
              
              
                VGG16 uses dropout layers, which will be disabled when model.eval() is used and which explains the difference during the model.train() comparison.