UPDATE : I executed the same code mentioned above, but on a different GPU, Tesla T4 (around 320 Tensor cores). There seems to be much of an improvement in the execution time, with and without amp
Without amp -
execution time in milliseconds: 3.9147518157958983

With amp -
execution time in milliseconds: 3.4673088073730467

The execution time with autocasting is slightly better than the one without autocasting. However, the time difference is not that great as expected (atleast 2x speedup would have been preferable).
What can be the reason for this ?
Is there any bug in the code ?
Is the GPU ineffective ?
Is the resnet18 model too small and simple to show any significant execution time difference ?