You might solve the performance issue setting torch.backends.cudnn.benchmark=True, if your input sizes are static (i.e. they don’t change in each iteration).
Have a look at this issue 5 for more information.

This issue might be fixed by cudnn 7.5.1, which was shipped in pytorch 1.1.
Please give it a try