CUDA operations are executed asynchronously so you would have to synchronize the code via torch.cuda.synchronize() before starting and stopping the timer, otherwise you would profile e.g. the CUDA context creation, the kernel launches etc.
Also, you should add warmup iterations and measure the average time of multiple iterations.