Solved by maaft in post #7 
              
              
                Solution 
For anyone wondering: It was indeed a problem with the batchnorm layers. 
The pretrained model was trained on a slightly larger training set with, therefore, different statistics. 
During training, the BatchNorm layers adapt to the statistics of the now smaller training set. Therefore the …