(Sorry for the delay, I was away)
That is one way of doing it, but the classical approach is generally to freeze the low-level features (disable gradient computing & updating of weights) and only finetune the high-level ones with a lower learning-rate.