Dear Community,
i encountered some none-intuitive behaviour. I load a model, set it to evaluation mode and predict a single image using model(input). Then, I set the model to training mode and predict a single image again. (Note that there is no .backward performed, I even disabled requires_grad for all parameters). When I set the model to evaluation mode again, and predict the same image from before, I get a different result.
TL;DR: when you eval an input before and after a training where no weight updates have been performed, the prediction in the evaluation phase changes.
Is this a bug or am I missing something here? You can run this code to check it yourself:
from torchvision import models
import torch

def set_parameter_requires_grad(model):
    for param in model.parameters():
        param.requires_grad = False

if __name__ == "__main__":
    model = models.densenet121(pretrained=True)
    set_parameter_requires_grad(model)
    model.train()
    input_ = torch.zeros((1,3, 224, 224))
    some_var = model(input_)
    model.eval()
    eval_value = model(input_)
    model.train()
    another_variable = model(input_)
    model.eval()
    eval_value_2 = model(input_)

    print(eval_value[0,0:3])
    print(eval_value_2[0,0:3])

output:
tensor([-0.3295,  0.2167, -0.6806])
tensor([-0.5839,  0.4981, -0.4104])

Edit: It’s not a dropout issue, the dropout_rate was 0 all along.