CUDA operations are asynchronous, so you should add torch.cuda.synchronize() before starting and stopping the timer.
Also, use a few iterations to get a better estimates as well as some warm-up iterations before the profiling.