Unfortunately, there is not much you can do at the moment besides using torch.backends.cudnn.benchmark = True.
I profiled the calls and all options call into the cudnn path here 22.
Once cudnn provides faster kernels, you will see the potential speedup.