Thanks a bunch for the reply. I started with pre-trained model, tuning the classifier/FC layer first and then sequentially unfroze the resnet-34 and vgg16_bn for training. As I unfroze resnet-34 layers the model seems to overfit on training data. Not sure if this is a symptom that I am not accounting for. The proportion of images across all class are constant across train and validation.