Thanks for the question, I also have the same issues in one of my network.
Even though the network and subsequent training ‘seems’ working, the model is not learning.
Model
Generator from a GAN Training, I am trying to assign labels to the generated images. Added an extra layer for the classification. Thus we add a Linear layer for classification at the end over and top of the GAN Generator. GAN training was completed with very low loss for both the Discriminator and the Generator.
Two observations,

Over the iterations the loss is decreasing but never close to zero even after 100 iterations. However, if I train the same model without prior weights(random initialization) from the GAN model is learning pretty fast(almost in couple of iterations) - This is how I passively inferred about (NOT) learning.
There are two labels in my class, one of them is classified correctly, however, even after repeated attempts to teach the other other label, learning is sparse. According to me the learning should be faster now.

Any insights to whats going is appreciated.