It’s possible your network is mostly bandwidth bound rather than compute bound, in which case the 1/2X memory traffic for fp16 ops would allow a speedup.  It’s also probable that many Pytorch kernels compute internally in fp32 even if the input/output is fp16, so the lack of raw fp16 throughput isn’t a problem.  It’s also possible the same is true for cuda library calls (gemms/convolutions):  they may use fp32 compute internally for fp16 input/output, so the compute throughput is not worse than fp32 and the required bandwidth is reduced.