Thanks for your reply.
I have another doubt here. While fine-tuning, should we not use a smaller learning rate for the whole network (http://cs231n.github.io/transfer-learning/ 1.6k)? Here the example says us to fix the network and only learn the weights in the newly added layers? Which one in recommended?