Solution
For anyone wondering: It was indeed a problem with the batchnorm layers.
The pretrained model was trained on a slightly larger training set with, therefore, different statistics.
During training, the BatchNorm layers adapt to the statistics of the now smaller training set. Therefore the error on the validation set is different than before.
The error will furthermore change from epoch to epoch because the trainloader is augmenting its data randomly.
Conclusion: When freezing a part of your model that is using BatchNorm layers, make sure to put them into eval() mode during training.