I am observing the same phenomenon running a Resnet50 network inference on a RTX3070:
without autocast: 11ms
with autocast: 16ms.
Interestingly the GPU power consumption and CPU utilization are both lower with autocast but the inference is just significantly slower on Pytorch 1.7.1.