I would recommend to use automatic mixed precision training via torch.cuda.amp 15 by installing the nightly binaries.
Simply transforming your model to float16 might easily create overflows/underflows and thus might yield NaNs.