CUDA operations are executed asynchronously, so you need to synchronize the code manually before starting and stopping the timers via torch.cuda.synchronize().
Based on your current description, you are “moving” the accumulated time from one operation to the next blocking one by commenting them out.
If you don’t synchronize the code, the next blocking operation will accumulate all previously executed (async) operations.