This might be due to the internal benchmarking of cuDNN, if you set torch.backends.cudnn.benachmark = True.
If your input size changes, cuDNN will run different algorithms and benchmark them to chose the most performant one for this particular shape.
Also note that you are missing a synchronization in the last part of your script.