Did you call model.eval() before performing the evaluation? This would make sure to e.g. use the calculated estimates in nn.BatchNorm layers instead of the batch statistics and disable nn.Dropout.
Have you trained the freezer parts before? Even if you freeze them by setting requires_grad=False they might still change if you use weight decay or if the optimizer uses some momentum or running stats.