The number of layers you are training in case of fine-tuning depends on the amount of data you have and also how similar the new dataset is to the one used for pretraining (ImageNet in case of torchvision).
Have a look at the Fine-tuning notes of CS231n 9 for a good overview.