Let me understand the issue properly.
You have two models, each with the same pre-trained base model.
Comparing the pre-trained section yields exactly the same results.
Maybe a stupid question, but when you pass the random input, is it the same input for both networks?
Were the models in training mode when you passed the inputs?
After this test (which gives the same result for the random input), you switched both models to evaluation mode and the results differ, right?
Did you compare the running_mean and running_var after the test failed already?