CUDA operations are asynchronously executed so you would need to synchronize your code before starting and stopping the timers via torch.cuda.synchronize() to get proper timings. Also, the operations could be executed in a loop to calculate the average time and reduce the noise a bit.
That being said, if your workload is small you won’t be able to saturate the GPU (and would also have the kernel launch overheads) and the CPU might be indeed faster.