Thanks for sharing this output!
I was able to reproduce a minor regression in the 1.10.0+cu113 wheels, which seems to be resolved already using the latest cuDNN release (8.3.0.96).
Code used to profile the workload:
random.seed(2021)
np.random.seed(2021)
torch.manual_seed(2021)

print(torch.cuda.get_device_name(0))
torch.backends.cudnn.benchmark = False

model = models.resnet50()
if torch.cuda.is_available():
    model.cuda()

optimizer = optim.Adam(model.parameters(), lr=1e-4)

if __name__ == '__main__':
    print("---start training...")

    # torch.backends.cudnn.benchmark = True  # use this for faster training
    torch.backends.cudnn.deterministic = False  # use this for more deterministic training
        
    inputs = torch.randn(32, 3, 224, 224).cuda()
    labels = torch.randint(1000, size=(32,)).cuda()

    # warmup
    for i in range(10): 
        # zero the parameter gradients
        optimizer.zero_grad()
    
        # forward + backward + optimize
        out = model(inputs)
        loss = F.cross_entropy(out, labels)
    
        loss.backward()
        optimizer.step()
    
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    for i in range(100): 
        # zero the parameter gradients
        optimizer.zero_grad()
    
        # forward + backward + optimize
        out = model(inputs)
        loss = F.cross_entropy(out, labels)
    
        loss.backward()
        optimizer.step()
    torch.cuda.synchronize()
    t1 = time.perf_counter()

    print((t1 - t0)/100)

Output:
1.8.2+cu102
deterministic = True
0.13085028601810336

deterministic = False
0.12192453876137734

benchmark = True
0.11950120938941837


1.10.0+cu113
deterministic = True
0.16201625971123576

deterministic = False
0.15164822606369854

benchmark = True
0.13653683779761194


source build + cuDNN8.3.0.96
deterministic = True
0.14494292575865983

deterministic = False
0.11826244482770562

benchmark = True
0.11710769269615412

You could either build from source using the current cuDNN release or wait until the wheels will be updated.