Have you try to using:

param.require_grad = False

Is similar to finetuning, but with the last layer fixed.
http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#finetuning-the-convnet 1.4k