I’m still unsure, how you are profiling the code, but note that CUDA operations are executed asynchronously, so you would have to synchronize the code via torch.cuda.synchronize() before starting and stopping the timers.
To profile certain operations, you could use the torch.utils.benchmark utilities or have a look at this post 3 to profile the code with Nsight Systems to isolate the bottlenecks.