CUDA operations are executed asynchronously and based on the description of the issue I guess you might not synchronize the code before starting and stopping the timers, which would return a wrong timing profile (e.g. only the kernel launches might be profiled).
Synchronize the code via torch.cuda.synchronize() before starting/stopping the timers, add warmup iterations, and in the best case calculate the average time of multiple iterations to stabilize the profiling.