Yes, it’s possible to use just parts of your model in FP16. Autograd will take care of transforming all gradients to the necessary dtype.
Also, if you are using cuDNN, some layers also take mixed precision input, so that you don’t have to specifically cast them to FP16 or FP32.
Have a look at NVIDIA’s apex 6 for some mixed precision utilities as well as this post 20 for more information regarding apex.