Since CUDA operations are asynchronous, you would have to synchronize your code before starting and stopping the timer via torch.cuda.synchronize().
Could you add it and profile the code again, please?