Hi Alex, This may be a good place to read up on Transfer Learning 5.
The imagenet pretrained network’s final output are the class log probabilities of the 1000 classes present in imagenet. Hence, it is possible that that particular final layer is in a deep local minima and may be hard to escape from. Effectively, the classification you were performing looks like
P(face expression| face image) = P(imagenet class| face image) * P(face expression|imagenet classes)