Solved by mcarilli in post #2 
              
              
                It’s possible your network is mostly bandwidth bound rather than compute bound, in which case the 1/2X memory traffic for fp16 ops would allow a speedup.  It’s also probable that many Pytorch kernels compute internally in fp32 even if the input/output is fp16, so the lack of raw fp16 throughput isn’…