Hi,
This happens because the CUDA api is asynchronous. So the first iteration, what you measure is just the time to queue work on the GPU.
After a while, that queue is full and you have to wait for stuff to be actually done before being able to enqueue more work.
You can use the equivalent of torch.cuda.synchronize() (torch::cuda::synchronize() maybe? not sure about the cpp API) to wait for all work to be finished before measuring time.