I’m not deeply familiar with the Tegra family. If you don’t have TensorCores, then the benefit would be the lower memory footprint, which might be used to increase the batch size and thus the throughput.
However, depending on your use case, this might not be interesting (e.g. for an inference use case with a single sample).
If you are using cudnn, you might want to activate torch.backends.cudnn.benchmark = True at the beginning of the script to use the cudnn heuristics, which should select the fastest kernels.
Note that the first iteration for each new input shape would be slower due to the benchmarking.
Let me know, if this reduces the FP16 latency.