Thank you @ptrblck, Yes I am doing exactly what you said. Using the first two layers of Model1 and adding a linear layer on top of it.
I have another question for the same use case. So, when I train my model for rotation prediction task with Adam optimizer, lr = 0.001, it gives less accuracy around 40%. But, if I change the learning rate to 0.0001, accuracy increases to 84%.
Is this normal for the model to behave like this or there is some bug in my code that I am not able to figure out?
Thanks in advance.