CUDA operations are executed asynchronously, so you would need to synchronize the code manually via torch.cuda.synchronize() before starting and stopping the timers or use e.g. torch.utils.benchmark, which will synchronize for you and will also add warmup iterations for a valid profiling.
Without syncs you would accumulate the duration of previous operations into a blocking one and would thus get invalid times.