Can you show your model? How are you tweaking it to your particular application? Usually, people replace the last FC layer with their own layer enabling the model to learn the low-level features specific to the particular dataset. Try replacing your FC layer and then train it.