I tried two methods for MNIST images generated by copying them 3 times.

Just change the output layers but freeze all previous layers
Change the output layer and fine-tune all other layers

Additional fine-tuning gives an about 4% accuracy boost, from around 95% to 99% for individual category.