Just to add some remarks on the benchmark trick here, for the convenience of further readers.
On my program I cannot see performance increase by setting torch.backend.cudnn.benchmark = True, even though the dimension of my input is always the same. I will need to investigate this later. But still running on GPU is a lot faster than running on CPU (GPU MX150 with 2G memory, CPU 8th gen i5, 8G memory).
Many thanks to ptrblck for answering my question.