@ptrblck Thank you for the suggestion. Infact, I did want to do what you are suggesting but couldnt find a way to do it: i.e how do I get the output of the penultimate layer and skip the classifier in the pretrained model? I can then attach my layers to this penultimate layer.
I would be really greatful if you could point me to any info on how to get the feature output from the penultimate layer.