@ptrblck thanks for your answer.
I have tried using different batch sizes e.g (8, 16, 64, 128). But I am not finding any difference.
Regarding the code, I am following examples here: https://pytorch.org/docs/stable/notes/amp_examples.html 3
Update: I am able to see the difference in inference time on GPU using
with autocast():
        with torch.no_grad():
            outputs = model(**inputs)

But when I compare the inference time on CPU, I do not notice any difference.