CUDA operations are executed asynchronously and based on your description and code snippet it seems you are not synchronizing the code, which could yield wrong results (and e.g. only profile the kernel launches until a sync is automatically added).
Add torch.cuda.synchronize() before starting and stopping the timer and compare the results again.