Ramesh_Kumar:

But when I compare the inference time on CPU, I do not notice any difference.


Automatic mixed precision is implemented for CUDA operations (and is thus in the torch.cuda namespace). By appying amp your GPU could use TensorCores for certain operations, which would yield a speedup. I don’t know if anything like that is implemented for CPU operations (and if I’m not mistaken not all operations are implemented for HalfTensors on the CPU).