I experienced a similar phenomenon when running very small networks with amp.autocast. In that case, pure fp16 inference was the fastest followed by fp32 and amp.autocast was the slowest.
Autocasting requires type conversion of input and the corresponding layers before the main computation. Type casting could take more time than the saved time from main computation in lower precision. In larger networks, the gains in computation will be more than the loss from the type conversion.