If you are using batch norm layers in the base model, the running estimates will still be updated even if you’ve frozen the affine parameters.
To fix the running stats, you would have to call .eval() on all batch norm layers.
Also, dropout layers might still be active, which could explain the different results.
You could also call .eval() on all dropout layers or alternatively on the self.base_model to disable these effects.