Based on this description I think you might not synchronize the code properly and your profiling might be wrong. CUDA operations are executed asynchronously so that you would have to synchronize the code via torch.cuda.synchronize() before starting and stopping each timer. Otherwise blocking operations would accumulate the execution times from all previous kernels.