Using float16 manually might be a bit tricky, as your parameters and activations might under/overflow.
If your model is working fine in float32, I would recommend to check out our native mixed-precision training via torch.cuda.amp 7.