From what I recall, you want to disable the built-in fp16 ops for PyTorch, something like
-DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__

or somesuch.
Quite likely the flags PyTorch itself builds with are indicative for what you need.
If your search the github issues for the flags, you might find some more background. I seem to recall that there was some issue with automatic casts or so.
Best regards
Thomas