Sorry for taking long to reply, I have been busy with some real life stuff.

You have replaced only 6th layer of classifier, so it looks like this:

It is true, the 6th layer of the classifier is not as supposed to be. The funny thing is that I have previously tried using the exact same layer you proposed, but it had the same results.

How many epochs did you run? Is the loss decreasing?

I did run about 40 epochs. The loss stays pretty much the same, if I remember correctly, and the validation accuracy behaves equally.

For fine tuning you can also freeze weights of feature extractor, and retrain only the classifier. You can also experiment with retraining only some layers of classifier, or whole classifier and part of feature extractor. You can do it like this:

I have not tried training only the classifier, but I suspect this would not change the results. To be honest, at this very moment I’m suspecting these problems are more related to the dataset itself than anything else. In the following days I’ll getting back into this project and I’ll be reviewing the dataset as a whole. I hope I can bring better news at the time.