@ptrblck your solution solved it …
Although I had to manually set trimmed_cnn to eval() after every evaluation since calling self.net.train() is recursive and sets trimmed_cnn to train mode.
    self.net.eval()

    # do evaluation ...

    self.net.train()
    self.net.trimmed_cnn.eval()  # this line is required... or the problem persist

Isn’t it weird BatchNorm layers are being updated if I’ve set requires_grad = False for all the layers?