CUDA calls are asynchronous, so if you would like to time them, you should call torch.cuda.cynchronize() before starting and stopping the timer.
Besides that, the first loop might be slower, because the DataLoader has to load the initial batch which might take some time. The following batches might be already waiting in the queue.