I tried to reproduce it with MNIST and ResNet18 as well as ResNet101 and see similar results:
ResNet18 time for just one epoch:

vanilla: ~38s

torch.backends.cudnn.deterministic = True: ~17s

torch.backends.cudnn.benchmark = True: ~13s

ResNet101 time for one epoch:

vanilla: ~77s

torch.backends.cudnn.deterministic = True: ~61s

torch.backends.cudnn.benchmark = True: ~52s

I would also expect the deterministic kernels to run slower.
Specs:

GTX1080 TI
Driver: 410.78
CUDA10.0

CC @ngimel is this expected or are we missing something?