I’m having similar issue with pytorch 1.7 w/ CUDA 11.0 compared to CUDA 10.1. I’m using 2080Ti as GPU.
Simple example that demonstrates this (only conv2d):
import torch
import torch.nn.functional as F

x = torch.randn(10, 64, 128, 128).cuda()
w = torch.randn(64, 64, 5, 5).cuda()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

# warmup
y = []
for _ in range(10):
    y.append(F.conv2d(torch.randn_like(x), w))

# measure
start.record()
y = []
for _ in range(10):
    y.append(F.conv2d(torch.randn_like(x), w))
end.record()
torch.cuda.synchronize()
print('time = %.2f' % (start.elapsed_time(end),))

Results:
# pytorch 1.7 w/ cuda 10.1
# time = 21.05 +/- 0.05
# pytorch 1.7 w/ cuda 11.0
# time = 25.40 +/- 0.05