It does look unexpected.
One comment I would make is that you most likely have bias in your convolutions? Did you forgot to freeze them as well?
So is your problem that your model is very severely overfiting to the training set?