Thanks for the update!
I’ve added some missing warmup iterations, since you are using cuDNN’s benchmark mode, so the first iteration will see a slowdown due to the internal profiling.
I can reproduce the slower kernel execution time in FP16 using the default memory format, but see a speedup using memory_format=torch.channels_last, which will avoid the internal permutations via:
cudnn::ops::nchwToNhwcKernel
cudnn::ops::nhwcToNchwKernel

Used code:
import torch
import torch.nn as nn
import time

torch.backends.cudnn.benchmark = True

conv = nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1, bias=False).cuda()
x = torch.randn(64, 3, 224, 224).cuda()


# warmup
for _ in range(10):
    out = conv(x)
    out.backward(torch.rand_like(out))
grad = torch.rand_like(out)

nb_iters = 100
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    out = conv(x)
    out.backward(grad)
torch.cuda.synchronize()
t1 = time.perf_counter()
print('{} s/iter'.format((t1 - t0)/nb_iters))


# warmup
conv.to(memory_format=torch.channels_last)
x = x.to(memory_format=torch.channels_last)
for _ in range(10):
    with torch.cuda.amp.autocast():
        out = conv(x)
        out.backward(torch.rand_like(out))
grad = torch.rand_like(out)

torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    with torch.cuda.amp.autocast():
        out = conv(x)
        out.backward(grad)
torch.cuda.synchronize()
t1 = time.perf_counter()
print('{} s/iter'.format((t1 - t0)/nb_iters))

Default memory format:
0.0012196038290858268 s/iter
0.0014858345501124858 s/iter

Channels-last:
0.0012191115505993367 s/iter
0.0008443104848265648 s/iter