You can change the nature of your tensor when you want, using my_tensor.half() or my_tensor.float(), my instincts would tell me to use the whole network with floats and to just change the output into half at the very last time in order to compute the loss.