Two quick comments:

The code does not take the async nature of CUDA into account, this is also the cause of your observation about GPU utilization. You would need to add torch.cuda.synchronize() before every time taking to make sure you measure the time when all preceding GPU work has been done.
It is preferable to use a monotonic clock (time.perf_counter()) for performance measurements,

Best regards
Thomas