The execution time highly depends on the used algorithm. E.g. you could see worse performance using cudnn for atypical input and kernel shapes. You could use torch.backends.cudnn.benchmark=True to call into the cudnn heuristics and execute a benchmark run in your first iteration to select the fastest kernels.


Getting stable results is not that easy and I would generally recommend to:



run a few warmup iterations right before the synchronization and the actual profiling
use a few iterations and calculate the average instead of isolated runs
lock the GPU clock, is you want stable results

Also, in case you are interested on some background knowledge, have a look at these GTC2019 slides 11 for best practices when benchmarking CUDA applications.