Could you share the code you’ve used to profile the code?
Note that CUDA operations are executed asynchronously, so you would have to synchronize the timer via torch.cuda.synchronize() before starting and stopping the timers.