I cannot reproduce the issue and get quite noisy results for the GPU workload e.g.:
0.0476837158203125
0.11849403381347656
0.06389617919921875
0.053882598876953125
0.05459785461425781
----
0.07605552673339844
0.15306472778320312
0.10085105895996094
0.08416175842285156
0.08344650268554688

Use torch.utils.benchmark to profile specific operations, as it would add warmup iterations and stabilize the timing using multiple iterations. Alternatively, use a profiler to check the actual kernel runtimes and check if your GPU goes into IDLE mode if it’s not used (which is why warmup iters are used during profiling).